I wonder if people are using a shared understanding of the word "understand." Most people taint the problem by conflating understanding with consciousness. When someone throws something at you, you don't "decide" to duck; you just do it, because you "understand" - in the way I would argue ChatGPT "understands" - that ducking will avoid harm. We call it a reflex, but there is no denying the logic of the response. You are not conscious of the decision to duck, just of the fact you have ducked. ChatGPT, I would say, is not "conscious" that it has reasoned something out, but it has reasoned it out.

I think of consciousness as the video RAM portion of my computer's memory. Lots of "thinking" goes on in the background of which I am not aware, because I choose to be aware of something else. But the thinking is going on. ChatGPT is like a computer with no video RAM. It has an operator, who feeds it questions from a customer. The program works entirely in background, spitting out outputs that make perfect sense to the customer but are invisible to the operator. Thus, the Operator/Computer "system" is capable of reasoning but is not conscious of what it has reasoned, i.e., does not understand what it has done, as that word is commonly used.