Alien Understanding

As I’ve been playing with ChatGPT, I’ve also been reading about Large Language Model AIs. It’s clear to me that ChatGPT doesn’t understand chemistry in the same way humans do, nor should I expect it to given what it is programmed to output; but does it understand language? I suppose that depends on what one means by “understand” and what one means by “language”. The distinction that comes to my mind is between syntax and semantics. My opinion is that ChatGPT mimics a human speaker because it is well-trained on syntax. We humans feel that it is an effective mimic because in a conversation, we have evolved to impute semantic information to any conversation partners, human, animal, or machine.

 

A useful primer that’s accessible to the non-expert is a recent short four-page article by Melanie Mitchell and David Krakauer. (“The debate over understanding in AI’s large language models”, DOI: 10.1073/pnas.2215907120.) Here are some of their statements that I felt clarified the key questions or highlighted the key issues.

 

“… the current debate suggests a fascinating divergence in how to think about understanding in intelligent systems, in particular the contrast between mental models that rely on statistical correlations and those that rely on causal mechanisms.”

 

“[In the past], the oft-noted brittleness of these AI systems – their unpredictable errors and lack of robust generalization abilities – are key indicators of their lack of understanding… However, [present LLMs]… can produce astonishingly humanlike text, conversation, and in some cases, what seems like human reasoning abilities, even though the models were not explicitly trained to reason.”

 

“… such networks improve significantly as their number of parameters and size of training corpora are scaled up… [some claim] will lead to human-level intelligence and understanding, given sufficiently large networks and training datasets.”

 

“…[others argue that LLMs] cannot possess understanding because they have no experience of mental models of the world; their training in predicting words in vast collections of text has taught them the form of language but not the meaning.”

 

“… the Eliza effect named after the 1960s chatbot… refers to our human tendency to attribute understanding and agency to machines with even the faintest hint of humanlike language or behavior.”

 

The crux is that we don’t exactly know, down to a nuts-and-bolts level (or should that be bits-and-bytes), what human understanding is. We have mental models – conceptual, abstract, and tied together by particular notions of causality – and we perhaps assume that such models aren’t solely statistical correlations. In this model, we have a self. And this self needs to function in the real world of blood-and-guts and nuts-and-bolts. As we interact with our environment, which may include other entities like ourselves, we use what we have learned to make predictions of what might happen next. We have expectations based on our conceptual models, and we are surprised when reality doesn’t meet expectation in a particular functional test. Then we update our model.

 

One argument against LLMs having human-like understanding is that they “exhibit extraordinary formal linguistic competence… they still lack the conceptual understanding needed for humanlike functional language… and use language in the real world. An interesting parallel can be made between this kind of functional understanding and the success of formal mathematical techniques applied to physical theories. For example, a long-standing criticism of quantum mechanics is that it provides an effective means of calculation without providing conceptual understanding.” As someone who teaches quantum mechanics, this analogy strikes home for me. But maybe it’s our present lack of understanding of our own understanding that’s the roadblock. And if we can learn how LLMs “understand” in their own way, that might shed more light on what human understanding means. It’s unclear that the lack of embodiment in present LLMs constitutes a significant hurdle, but one could conceive outfitting a physically mobile robot with an LLM. Such an integration could prove interesting. And scary.

 

If future improved LLMs cannot create something that resembles a rich-enough mental model that seems akin to humans, can they at least do what is functionally equivalent? Mitchell and Krakauer suggest that when we see “large systems of statistical correlations produce abilities that are functionally equivalent to human understanding… [or] enable new forms of higher-order logic that humans are incapable of accessing… will it still make sense to call such correlations ‘spurious’ or the resulting solutions ‘shortcuts’? And would it make sense to see the systems’ behavior not as ‘competence without comprehension’ but as a new, nonhuman form of understanding?” They also caution us not to jump to conclusions when trying to interpret how LLMs perform on our “tests” of comprehension based on our own assumptions of how human understanding works – something we don’t really understand at a deep level. It’s perhaps an alien understanding. We might anthropomorphize such alien understanding in sci-fi literature and media, but perhaps that’s the only way we as humans can try to understanding something beyond our ken. If anything, all this reminds me that we need to remain humble about what we think we know. We’re limited by our own human embodied understanding.