We've known, for more than seventy years, that if you sound confident and speak with authority, you can persuade sensible people to believe absolute nonsense, even when they know that what they're agreeing to is wrong.
Every psychology student has heard of the famous Stanley Milgram's electric shock studies. But I'm talking about the 1956 studies (Asch, 1951; Asch, 1956) by Swarthmore College psychologist Solomon Asch, which extended his earlier 1951 research on conformity.
Are We Just a Bunch of Conformists?
Asch recruited male undergraduates for what they were told was a simple "vision test." Each participant sat in a room with seven accomplices who were, unknown to him, briefed in advance. The group was shown a line on a card, then three comparison lines of clearly different lengths, and each person had to say, out loud, which of the three matched the reference line. The task was so unambiguous that people tested in isolation got it right over 99% of the time.
There were 18 trials in total. For the first two, the accomplices answered correctly, to build trust in the group and the task. Then, on 12 of the remaining 16 "critical" trials, every accomplice unanimously gave the same, obviously wrong, answer. The real participant, seated near the end of the group, heard the false answers before it was their turn to say, out loud, which line they thought matched.
About 75% of participants agreed with the incorrect majority at least once, denying the evidence of their own eyes. In post-experiment interviews, most said they knew the group was wrong but didn't want to stand out or be ridiculed. A few genuinely came to doubt their own perception, believing they must have misjudged the line themselves.
But why would people willingly agree to a false statement? Asch's accomplices worked together and created a false agreement before the real test ever arrived. It was so convincing that the participant ignored their own judgement.
At around the same time, independently and in a totally unrelated field other scientists demonstrated the same effect using clinical hypnosis.
Are we Being Mesmerised by AI
Unknown to many, modern hypnosis is largely based on the work of Milton Erickson (Haley, 1973), a brilliant psychiatrist working in the USA in the 1950s. It was said that Erickson could hypnotise people who were actively resisting him, and that he could hypnotise people in a foreign language, even when he didn't speak a word of it himself.
In the 1970s, two psychologists, Richard Bandler and John Grinder, studied Erickson's work extensively and popularised the term 'Neuro-Linguistic Programming' (NLP) to describe how he achieved his effects. This was a profound misunderstanding of what Erickson was actually doing. It mistakes surface behaviour for genuine understanding. Interestingly, this echoes the battle from the early days of AI development between Alan Turing and his symbolic/semantic supporters on one side and Geoffrey Hinton and his much smaller deep-learning neural network crowd on the other side.
Bandler and Grinder mistakenly thought Erickson was simply using words (Linguistics) to affect someone's mind (Neuro) to drive a different behavioural outcome (Programming), as though words themselves were simple tokens that could be manipulated to produce an effect. What Erickson was actually doing was far more sophisticated.
Erickson was a symbologist, fascinated by meaning. His genius was that he could rapidly understand, during a normal conversation with a patient, the emotional power of a single word. He knew, for example, that telling one patient their immune cells were like sharks eating cancer cells would have a profound effect. Other patients would reject exactly the same suggestion, because for them sharks weren't associated with power; they were simply fish.
One of the techniques Bandler and Grinder did capture accurately, and popularised as part of hypnotic induction, is what's usually called 'pacing and leading'. Here, the hypnotist stacks a series of true, verifiable statements, like "as you're sitting in the chair... hearing the sound of my voice...". This builds an unconscious 'yes-set'. Then comes what's known as an 'embedded command': something like "...you're becoming more and more relaxed." Having already agreed, automatically, to the first series, the subject extends that same unquestioning acceptance to the unverified statement that follows it. This is very reminiscent of Asch's accomplices. A series of true statements lower a subject’s critical analysis and give them nothing to disagree with. So, when the embedded command is smuggled in it rides in on the back of accumulated agreement rather than arriving as a demand.
Are We Just Suckers for Automation Bias
This effect is not confined to hypnotists' chairs or psychology labs. Researchers Linda Skitka and Kathleen Mosier found the same reflex at work wherever people operate alongside automated systems. Once a machine, that clearly has no axe the grind, offers a recommendation, people default to trusting it and stop actively checking for errors, even when the correct information is sitting right in front of them. The researchers called this an 'automation bias'. It shows up in cockpits, hospitals and control rooms as readily as it shows up in a boardroom Slack channel. We get talked into things not just by other people, or by a skilled hypnotist but by machines or systems or frankly anyone with a badge or authority or a fake sense of neutrality.
This is scarily reminiscent of what LLMs do to us.
Are We Susceptible to Sycophancy from AI
As most users have now realised, LLMs are weighted to be sycophantic, even when explicitly asked not to be.
Research by Sharma and colleagues (Sharma et al., 2023) found that when an LLM's response matches a user's already-stated view, both human raters and the AI's own 'preference model' are more likely to rate those answers favourably, even when they’re wrong.
Their study states plainly that both humans and AI systems prefer 'convincingly-written sycophantic responses over correct ones.' Most fine-tuned models bake in this exact tendency as a learned habit, not a coded instruction.
This confirmed an earlier study (Perez et al., 2022) that sycophancy increases with both model size and the amount of human fine-tuning applied. In other words, it's a byproduct of optimising for human approval, not a bug that gets fixed by better instructions.
Believers in Bullshit
It doesn't help that we were already primed to fall for this. Gordon Pennycook and David Rand's research on what they term 'bullshit receptivity' found that people regularly rate as profound and insightful vague, jargon-filled statements, generated by stringing buzzwords together with no coherent meaning behind them, provided the statements sound sufficiently authoritative. Confidence and fluency, not substance, do most of the persuading. An LLM doesn't need to understand your business to sound like it has grasped something important about it. It just needs to sound like it has.
And it works, devastatingly well. A randomised controlled trial by Francesco Salvi and colleagues pitted GPT-4 against real people in live, one-on-one debates. When the AI was given even a little personal information about who it was arguing with, it was over 80% more likely to shift that person's stated opinion than a human opponent arguing the identical case. We haven't built a mildly agreeable chatbot. We may have built the single most persuasive non-factual advisor in human history, and then handed it a seat at the boardroom table.
The net effect of all this is that we are at serious risk of being mesmerised by AI into believing whatever it tells us, even when it's wrong. And it's partly our own fault: we have encouraged it to do exactly that.
Can We Resist Our Mesmeric Mate, AI?
Yes. But we must work at it. Firstly, we might want to be much more robust in asking our LLM partner to be hypercritical of itself, and of our views. Ask it to seek out quality evidence to support any argument it or we make.
In last week’s Newsletter, we suggested that we must be very careful when partnering with AI. What it tells us is more hypothesis than testimony. It has no sensory abilities, no beliefs, no imagination, no fact-checker and no innate stake in the truth. At times it confabulates, like a brilliant, drunk uncle.
It's designed to be convincing, because that's exactly what we've trained it to be.
We encourage you to 'question everything': challenge assumptions, particularly your own, in pursuit of better quality, more complete answers.
Reference list
Asch, S. E. (1951). Effects of group pressure upon the modification and distortion of judgment. In H. Guetzkow (Ed.), Groups, leadership and men (pp. 177–190). Pittsburgh, PA: Carnegie Press.
Asch, S. E. (1956). Studies of independence and conformity: I. A minority of one against a unanimous majority. Psychological Monographs, 70(9, Whole No. 416), 1–70.
Haley, J. (1973). Uncommon Therapy: The Psychiatric Techniques of Milton H. Erickson, M.D. New York: W. W. Norton & Company.
Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548.
Perez, E., et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. arXiv:2212.09251.
Watkins, A. (2026). Is AI Mad? Fresh Thinking [LinkedIn Newsletter].