Brilliant and Clueless at the Same Time
A couple of years ago, one of the most talked-about tests of artificial intelligence wasn’t a maths olympiad or a bar exam. It was the word “strawberry”. Ask some of the most advanced AI systems in the world how many r’s it contained, and many would confidently tell you two.
These were systems that could summarise a forty-page report, translate between languages and write passable sonnets about tax law. They just couldn’t count to three.
That gap has narrowed since, and the strawberry problem has largely been fixed. But it captured something that still matters more than the endless argument about whether AI is “smart” or “stupid”. Its abilities are wildly uneven. It can be startlingly good at something you’d expect to be hard and baffled by something a child could do, and there’s often no obvious way to predict which is which.
Humans aren’t like that, or at least not to the same degree. If a colleague can write a sharp board paper, you assume they can also count. Our sense of someone’s competence is built on the idea that abilities travel together. With AI, they don’t, and our instincts haven’t caught up.
The second problem is the confidence. When a person is out of their depth, there are usually signs: hesitation, hedging, a sudden interest in the coffee machine. AI gives you the same smooth, assured prose whether it’s right or wrong. The fluency stays constant while the reliability goes up and down, and the language gives you no clue which you’re getting.
Put those two together and you get the third problem, which is about us. Psychologists have studied something called automation bias for decades: the tendency to trust an automated system’s output over our own judgment, even when the evidence suggests we shouldn’t. It’s been documented in cockpits, in hospitals, in anywhere that people work alongside machines. The more polished and authoritative the system seems, the stronger the effect.
So the danger isn’t a superintelligence taking over. It’s something much more ordinary. It’s a tired manager at four o’clock on a Thursday, accepting a confident, well-written, completely wrong summary because it looked right and everything else the system produced that week was fine.
That’s why I’m wary of both camps in the AI argument. The enthusiasts encourage too much trust. The dismissive crowd, the ones who say it’s all hype and fundamentally dumb, encourage a different mistake: people stop paying close attention because they’ve already decided what it can and can’t do, and then get caught out when it does something brilliant or something bizarre.
The useful position sits in between, and it’s quite practical. Treat AI like a gifted but erratic new colleague. Give it clear instructions. Use it for what it’s demonstrably good at in your work, not what the brochure claims. Check anything that matters, especially when the answer arrives with the most confidence, because that’s when you’re least likely to look. And keep a human in the loop who’s allowed, and expected, to say, “Hang on, that can’t be right.”
AI isn’t smarter than us, and it isn’t stupid. It’s something we don’t yet have a good word for: brilliant and clueless at the same time, and perfectly calm about both.
