The room was full of some of Britatain’s most eminent leaders; the Chairs and CEOs of the FTSE 100.
The debate was lively and animated. After about an hour the facilitator came to me and asked:
“As a neuroscientist, what’s your view on what people have been saying this morning”
I said: “The conversation has been very interesting. But it’s largely been focused on ‘doing the undoable’. People have shared stories about what their company is doing about the various commercial and geopolitical challenges we’re all facing. We’ve heard CEOs suggest what their companies could DO differently, and whetehr that would give them a strategic advantage.”
“But”, I continued, “it’s ironic that, in a conference entitled ‘Thinking the Unthinkable’, there’s one thing we’ve not thought about”
“What’s that”, asked the facilitator
“Thinking” I said.
“We’ve been talking about doing things. But we’ve not mentioned the word ‘thinking’ or even explored what a thought is. We’ve not discussed how thoughts arise. We’ve not discussed why thoughts arise or what determines the quality of our thinking. And we’ve certainly not discussed whether it’s possible to think an unthinkable thought”.
The room fell silent for a moment.
It felt a little like I’d just suggested the king had no clothes. The room was unsure how to respond to my challenge.
Within a couple of minutes, the debate in the room returned to a conversation about what each leader was doing and whether they could do different things.
One of the world’s greatest ever thinkers, Richard Feynman, tells a similar story. He recounted that when he was a boy his father took him into the Catskill mountains, pointed to a bird and said, look in Italian that bird is called a Chuto Lapitida; and in Portuguese it’s called a Bomba Peda and in Chinese it’s called a Chong Long Ta. But just because you know what that bird is called, in different parts of the world, doesn’t mean you know anything about the bird.
This was a breakthrough in the way he thought about the world because he realised that we can all repeat what we’ve been told in a classroom but that doesn’t mean we understand what anything means.
This difference between knowledge and understanding takes us to the heart of the AI debate as well as the philosophy of mind. It’s really the difference between symbology and semantics.
Symbology is the study of symbols, their design, their use, and their interpretation. Symbology is concerned with the shape and system of symbols. What icons do we use to represent things. And how are those representations organised.
Semantics is concerned with meaning in language. How do we use words, phrases, and sentences to convey meaning. How does meaning shift with a change in context. How does a change in the relationship between linguistic statements change what they refer to in the world.
Large language models (LLMs) use symbology not semantics. They learn statistical patterns that exist between the symbols in their vast training data sets. They have vast knowledge, but each individual piece of knowledge has zero meaning. Meaning can only be inferred from other symbols, not from experience.
In the AI world these symbols are called ‘tokens’ because the words in the training sets have been reduced to 1s and 0s, so they can be processed. AI can then produce outputs that look semantically rich, but the LLM has no grounding in the real world.
AI has no perception, no embodied experience of anything, and no reference point beyond its symbols. So, in a sense, it's symbols all the way down. This was called the symbol grounding problem by Stevan Harnad, in 1990. LLMs behave as though they understand something, but they don’t. They are ungrounded symbol systems infected with a severe case of classical “aboutism.”
Thus, LLMs know about mental health, but they have never experienced it. Even adding imagery or sound to LLMs, rather than just operating from text, doesn’t give them an experience, because the imagery and sound are also reduced to tokens.
What makes us uniquely human is we doesn’t just understand the word “fire” we have an experience of that word which may include the idea of danger, heat, warmth, light and so on.
In contrast, an LLM’s understanding of the word fire is anchored to other words about fire. AI has no experience to bring richness to the word.
In developing his own thinking Richard Feynmann described how would convert what he read about quantum physics into an experience, in his mind, of atoms and quantum equations.
Now here’s the grey area.
Most human being’s understanding isn’t generated from first-hand experience.
We tend to understand what things mean because someone’s told us what they mean.
We haven’t always experienced these things for ourselves. We haven’t interacted with our own thoughts and connected them to something real, as Feynman suggested. We haven’t tested their validity, explored their origin or challenged their usefulness.
The reason why most people don’t understand what anything means is because they haven’t ever learnt to think properly.
When most people offer you an explanation, they’re simply passing on what they have been told.
Very few people have ever thought about their own thinking.
They’ve never questioned whether their thinking is rooted in their own experience or whether they are just operating with hand-me-down thinking and meaning from others.
We’ve mostly been brainwashed or spoon fed what to think, and we’ve been told what things mean. Mostly people are just recycling.
This is not thinking at all, it’s repeating.
This debate goes right to the heart of whether AI is thinking or not , and whether we should use LLMs as a “thinking partner”.
AI is just doing a brilliant job at processing and predicting what we want to hear.
Since many people have now realised that AI can be sycophantic, they ask it to be extremely critical and robust in its feedback or challenge. They hope that by trying to reset its weighting with some simple prompt engineering they can avoid the risk of being stuck in an echo chamber of their own making with an obsequious toad rather than a genuine ‘thinking partner’.
This debate has roots in philosopher John Searle’s Chinese Room experiment, which he published in 1980. Searle, who died in 2025, suggested an experiment where a person, who doesn’t understand Chinese, sits in a room with a rulebook for manipulating Chinese characters. By following instructions, the person produces outputs that convince outsiders he understands the language, even though he does not. Searle’s Chinese Room experiment was challenging the validity of the Turing test.
The Turing test, which is a test of a computer’s human-like intelligence and ability to “think”, suggest that if we can fool people into believing that a computer’s response is indistinguishable from a genuine human response then the computer must be operating so close to human intelligence that the difference is meaningless.
The problem is that the difference isn’t meaningless.
The Turing test just shows that the computer can fake meaning so well that people can’t tell. But this doesn’t mean that the computer is thinking.
Imitation is not ideation.
AI may be able to process its tokens so well that it looks like it’s actually thinking, but that doesn’t mean it IS actually thinking.
If AI was capable of consistent ideation, we’d be a lot closer to artificial general intelligence (AGI) and, despite what Sam Altman claims, no AI or LLM is close to achieving AGI.
Until AI can think the unthinkable, which is a uniquely human skill, there is still room for people in our Agentic AI future.
Thinking is not something we’re born with, it’s a habit we build.
But, as Richard Feynman says, most people never develop their ability to think. Why? Because nobody teaches them how to think. If that’s true they also never develop their ability to think better quality thoughts or think the unthinkable.
Think about that!