Last week I was working on fine-tuning an AI model so it could rate the sophistication of a leader’s statements about business. I also asked it to explain its thinking.
A leader wrote “business must drive the betterment of society, and today that’s more important than ever as governments do less to improve society”.
AI rated the sentence and then explained its reasoning as: “business and society are framed as an integrated collective endeavour aimed at improving life 'for as many as possible'.”
I was amazed. The LLM just totally invented something that the leader never said, to justify itself .
I started checking AI’s justification of its scoring on other statements.
In one statement the leader suggested “a good boss is one that teaches you to be better and helps people build lives and not just careers”.
The LLM scored the sentence and explained the reasoning for its scoring by stating: “the description of a good boss as someone who must be feared, respected, liked, and loved is outcome focused. There’s humour about his wife being his boss who 'fuels him with great wine'”
Wow. This explanation was a much worse fabrication than the previous example.
Is AI Hallucinating?
I discussed this with our Chief Engineer who said: “Oh that’s just AI hallucinating”.
If one of my children were hallucinating, I’d take them to a psychiatrist, worried that they were in the early stages of a psychotic episode.
But somehow, we tolerate such madness from AI, and we often dismiss it as a quaint, almost cute, characteristic.
Many leaders and businesses are increasingly using AI to guide critical decisions, but we don’t seem very worried that this brilliant expert, who is influencing us, may actually be mad, intermittently hallucinating, or becoming psychotic.
Then I realised that this isn’t really an hallucination. Hallucinations require sensory data. In fact, the definition of an hallucination is a vivid and clear perception that has the impact of a true perception but isn’t anchored in the relevant sensory data.
Hallucinations, which are one of the key diagnostic features of schizophrenia, may be visual, auditory, olfactory, tactile or gustatory (sight, sound, smell, touch or taste), and auditory hallucinations are by far the most common in people with schizophrenia
An LLM has no sensory organs, no perceptual apparatus, no subjective experience at all. Therefore, its perception can’t be clear or clouded. Therefore, it’s wrong to suggest that AI is hallucinating. It’s also wrong to suggest that AI is experiencing an illusion. An illusion is a misperception of an actual stimulus. Again, this is not true for the same reason.
Is AI Deluded?
It’s not that AI is deluded. A delusion is a fixed belief that’s not amenable to change in light of conflicting evidence. The difference is that an hallucination is a disturbance of perception and a delusion is a disturbance of belief. A deluded person isn’t seeing something that isn’t there, they are resisting correction, even when presented with clear, agreed-upon contrary evidence. AI isn’t deluded because it will cheerfully reverse its view the moment you push back, correctly or not. There's no belief being protected because AI has no belief there to begin with.
Is AI a Fantasist?
This is a much closer description to what AI is doing than an hallucination. A fantasy doesn't claim to be true. It’s a normal imaginative mental activity, not a break with reality. But a fantasy still implies someone, in this case AI, is wishing for something because the fantasy serves a positive psychological function. But an LLM has no wishes, so can’t engage in fantasies.
Is AI engaging in ‘Magical Thinking’?
AI is not doing this either, although this gets closer to the truth than ‘hallucination’. Magical thinking is the belief that one's own thoughts, words, or actions can influence events in ways that violate normal cause and effect.
Magical thinking is normal in a three-year-old. They will often believe the “clouds are following me”. But when a psychiatrist sees an adult engaging in magical thinking, they often wonder if that adult has OCD or schizotypal personality disorder. Adult examples would be knocking on wood for good luck; or believing a thought, on its own, can cause harm or benefit; or believing a ritual can change the result, unrelated to any physical mechanism.
A leader who treats an LLM's fluent, confident output as inherently true because it's spoken with authority is engaging in something close to magical thinking themselves. They are granting causal, truth-bearing power to the AI’s words because of how they sound rather than whether they’re rooted in fact.
Is AI engaged in Mythical Thinking?
Mythical thinking is one developmental level up from magical thinking. Children who don’t understand how the world works believe it’s magic. As we grow up and start to understand more about the world, we make up stories (myths) to explain what happens.
If you imbue AI’s output with mythic power and believe its authority as true, simply because it was "spoken" fluently and confidently, and you don’t apply any mature critical analysis to what AI says or test its claims, then the failure isn't in AI, it's with you. You’ve bought into the mythology of AI itself, and you’ve now engaged in mythic thinking even though AI doesn’t.
So, AI is not hallucinating; it’s not deluded; it’s not fantasising or engaging in magical or mythic thinking. All five processes would require AI to perceive, believe, wish, or try to create meaning. AI has none of that architecture.
Is AI simply Lying or Bullshitting?
AI is not lying.
Lying requires three things: a false statement, an understanding that the statement is false, and an intent to deceive. A liar knows what’s true and they carefully steer you away from it. LLMs have no internal representation of "what's actually true" that they are knowingly contradicting.
In contrast to liars, who know what’s true, bullshitters are indifferent to truth. Bullshitters aim to produce an effect, an impression of authority, competence, or fluency, with no regard at all for whether the underlying statements are accurate. This is much closer to what an LLM is doing. LLM’s are a ‘next-token-prediction system’ optimised to produce plausible, fluent, contextually appropriate text. It has no internal mechanism that tracks or cares about truth as being important and different from plausibility.
A 2024 paper by Hicks, Humphries and Slater in Ethics and Information Technology, provocatively titled "ChatGPT is Bullshit," makes exactly this argument and has gained real traction in the AI ethics literature.
They suggest LLMs should be understood as bullshit machines, not as hallucinating minds, because "hallucination" smuggles in a false implication that the system has a perceptual or perhaps even a belief-forming relationship to reality that it is malfunctioning within, when the truth is that there was never a truth-tracking mechanism there to malfunction in the first place.
Is AI Confabulating like an Alcoholic?
Perhaps the most accurate description for what AI is actually doing is confabulating.
Confabulation is the generation of a fabricated, distorted, or misinterpreted memory or narrative, produced without any conscious intention to deceive. The confabulated statement typically fills a gap where there’s no information, but the confabulator continues to answer fluently.
Confabulation is common in individuals with Korsakoff's syndrome, a brain condition associated with chronic alcoholism due to thiamine deficiency. Confabulation can also occur in people with malnutrition, eating disorders, malabsorption and some cancers.
Korsakoff's syndrome involves real brain damage to the limbic system, thalamus, mammillary bodies and hypothalamus, all critical for memory and emotion.
Confabulation is a failure of the brain's normal "checking" or reality-monitoring process. Normally our brains would flag "I don't actually know this" before words come out. Confabulation is a breakdown in our fact-check, a loss of truth-tracking.
What is AI’s Madness then?
The best description of AI’s “madness” is confabulation rather than bullshit.
Bullshit statements aren't produced by a malfunctioning honesty-checker. They're produced by a process that was never oriented toward the truth in the first place.
A bullshitter’s goal is impact. It must sound right, land well, and move the conversation along. Nothing goes wrong when a bullshitter bullshits. The system does exactly what it's built to do.
Brilliant bullshit is close to what AI does. AI doesn’t have a truth-tracking mechanism that fails. Truth isn’t AI’s optimised target. At best truth is a loose correlate of the training data. AI’s tendency to bullshit is made worse by the fact that human fine-tuning of LLMs actively rewards confident, comprehensive-sounding answers over honest uncertainty, because humans prefer such answers.
Confabulation is the much more accurate term for what AI actually does because it comes from inside the machine rather than from an error in the philosophy of truth.
Anthropic's own research, published in 2025, called "On the Biology of a Large Language Model," traced an internal feature that operates a "do I actually know this?" circuit. Under normal operation, when the model doesn't recognise an entity or fact, this circuit suppresses the answer and triggers a refusal or hedge.
An AI “hallucination” occurs specifically when that suppression circuit misfires, when the model incorrectly registers "I know this" and proceeds to generate a fluent, specific, entirely fabricated answer anyway. The fact that AI’s checker misfires has been independently confirmed. AI confabulates because it has a structural monitoring failure, something inside the system is supposed to catch the gap and doesn't.
So, the most accurate statement of AI’s “madness” is that an LLM is a brilliant bullshit machine by design, which occasionally confabulates in practice like a chronic alcoholic.
What is worrying is that the industry's own metaphor exonerates the technology by implying an occasional glitch in an otherwise anthropomorphised truth-seeking system.
Suggesting that AI is “hallucinating” is such sloppy thinking that it may be cause leaders to misunderstand and mismanage the actual risk.
Leaders may picture the machine "seeing things," which it can’t because it has no perception.
They may also mistakenly believe that LLMs track the truth when they don’t. So, when AI’s own safety checks fail, they confabulate so fluently, and without any warning or awareness of their own uncertainty, we believe them.
So we must understand what this AI genius on our phone or laptop is doing, and more importantly not doing. We should not automatically trust what it says any more than believing the elaborate narrative of a chronic alcoholic. If we understand how AI really works we can partner with it much more effectively to create a brighter future for us all.