What AI Cannot Do

That’s not a real case. But it’s an entirely plausible one. And variations of this story are already happening — patients asking AI about symptoms, receiving reassuring answers, and delaying care that they needed urgently.

In the last lesson, we looked at the genuine opportunities. Now we need to look at where this technology falls down. Because the limitations aren’t obvious — and that’s precisely what makes them dangerous.

1. Hallucinations. Language models generate text based on patterns. Sometimes those patterns produce wrong answers — and the wrong answers sound exactly as confident as the right ones. Research shows 8–20% of medical responses contain inaccuracies. If a patient asks ten health questions over a month, one or two answers will contain something untrue. The patient has no way of knowing which ones.

2. Delayed care. When a patient gets a reassuring answer from AI, they feel reassured. But AI doesn’t examine patients. It doesn’t hear the slight catch in their voice. It doesn’t notice the ankle oedema they forgot to mention. It doesn’t see the subtle distress behind “I’m sure it’s nothing.” Every GP has had the patient who nearly didn’t come in. Now imagine that patient had an AI tool telling them their symptoms were probably benign.

3. Context confusion. The majority of AI training data comes from the United States. AI will refer to acetaminophen instead of paracetamol. It will give blood glucose values in mg/dL instead of mmol/L. It will suggest calling a “primary care physician” instead of your GP, or going to the “ER” instead of A&E. For a patient told their blood sugar of 7.8 is dangerously high because the AI assumed American units? That creates real confusion and real anxiety.

4. Missing red flags. AI processes text. It doesn’t have the pattern recognition that comes from years of seeing patients. A patient types “I’ve had a headache for two weeks and I feel a bit off.” A GP might think about temporal arteritis, raised intracranial pressure, carbon monoxide exposure. AI might suggest paracetamol and hydration. It doesn’t have the instinct that says something about this doesn’t feel right.

You might be thinking: if it’s this risky, should we just tell patients not to use it?

We can’t. And we shouldn’t try.

Think about how we approach alcohol. We know it causes harm. We know some people will use it irresponsibly. But prohibition doesn’t work. It just drives the behaviour underground.

40% of UK adults are already using generative AI. Telling them to stop isn’t realistic. What we can do is help them use it more safely. This is harm reduction, not prohibition.

It helps to think about AI use on a risk ladder:

The principle is simple: the closer the AI use gets to a decision — especially about treatment or whether to seek care — the higher the risk.

You will be asked about this. Here’s what we suggest:

Simple enough for a patient to remember. Nuanced enough to actually be useful.

If they want more detail: “It’s a great tool for understanding your condition better. For preparing questions before you see me. For translating medical jargon into plain English. But when it comes to symptoms, treatment decisions, or anything that worries you — come and talk to a real person.”

The danger is not that AI is terrible. It’s that it’s good enough to be trusted — but not reliable enough to be trusted blindly.

If AI were obviously bad, no one would use it for health questions. The problem is that it’s impressive most of the time — which makes the failures harder to spot.

Your role, as a clinician who now understands how this technology works, is to help your patients navigate this. Not to ban it. Not to ignore it. To guide them.