Despite their growing popularity, AI chatbots health advice may be unreliable and potentially dangerous, according to new research published on Monday.

While artificial intelligence systems can now perform exceptionally well on medical licensing exams, a study found that chatbots do not help people make better real-world health decisions compared with traditional online searches.

“Despite all the hype, AI just isn’t ready to take on the role of the physician,” said study co-author Rebecca Payne of Oxford University.

Study Tests Real-World Use, Not Exams

The British-led research team examined how well people could identify health problems and decide whether medical care was needed when using AI chatbots health advice.

Nearly 1,300 UK-based participants were given 10 everyday medical scenarios, including headaches after drinking alcohol, exhaustion in new mothers, and symptoms associated with gallstones.

Participants were randomly assigned to use one of three chatbots — OpenAI’s GPT-4o, Meta’s Llama 3, or Command R+ — while a control group relied on standard internet search engines.

Accuracy No Better Than Web Searches

The results showed that people using AI chatbots correctly identified their health condition only about one-third of the time. Just 45 percent were able to determine the correct next step, such as seeing a doctor or going to hospital.

According to the study, these outcomes were no better than those achieved by the control group using search engines. The findings were published in the journal Nature Medicine.

Communication Gaps Drive Poor Outcomes

Researchers attributed the weak performance of AI chatbots health advice to communication breakdowns between users and the systems.

Unlike simulated patient interactions used in benchmarks, real users often failed to provide all relevant details. In other cases, participants misunderstood chatbot responses or ignored the guidance altogether.

The study noted that one in six U.S. adults already consult AI chatbots for health information at least once a month, a figure expected to grow as adoption increases.

Experts Urge Caution

“This is a very important study as it highlights the real medical risks posed to the public by chatbots,” said David Shaw, a bioethicist at Maastricht University who was not involved in the research.

He advised people to rely on trusted medical sources, such as national health services, rather than depending on AI chatbots health advice for diagnosing symptoms or making urgent medical decisions.