Artificial intelligence (AI) is becoming an increasingly familiar presence in modern healthcare. From assisting with diagnostics to supporting treatment decisions, these sophisticated systems promise to transform medical care.

Yet, a new study published in PLOS Digital Health reveals a sobering reality. Physicians, like the rest of us, are not immune to the pitfalls of trusting technology—even when it is demonstrably wrong.

Researchers from a leading Spanish university set out to test how doctors interact with AI-generated advice, especially when the technology makes mistakes. The study involved 223 physicians who participated anonymously in controlled online experiments. Their mission was straightforward but carried profound implications for the future of medicine.

Doctors were asked to imagine treating patients with a rare disease. The twist? The proposed treatment was experimental and its effectiveness unclear. The AI system provided recommendations on which patients were more likely to benefit from the treatment. After each decision, doctors received feedback in the form of patient recovery data. The expectation? The trained professionals would use this data to verify the AI’s advice.

That’s not what happened.

In both experiments, the AI’s recommendations did not match reality. In one scenario, the treatment was moderately effective for all patients, regardless of what the AI suggested. In another, it was completely ineffective across the board. Despite clear evidence contradicting the AI’s advice, many doctors continued to trust the system and rated it as reliable. Even repeated exposure to recovery data showing no improvement failed to shake their confidence in the algorithms.

This phenomenon is not entirely new. Prior research has shown that people, including highly educated professionals, often struggle to detect and correct AI errors. What’s striking here is how this plays out among those tasked with life-and-death decisions.

In these experiments, even when doctors had all the evidence they needed to question the AI, they largely failed to do so.

Why does this happen? Trust in technology is a double-edged sword. On one hand, it enables clinicians to use powerful tools that can analyse vast datasets more quickly and efficiently than any human could hope to achieve. On the other hand, over-reliance can lead to missed errors and, ultimately, harm to patients.

Medical institutions and policy makers have been quick to champion AI as a solution to many of healthcare’s most pressing problems ranging from staff shortages to rising costs and diagnostic errors.

The narrative often goes, machines do not get tired or distracted; they process information without bias or emotion. But this study suggests that human users may actually overestimate these strengths, particularly when evidence points in another direction.

The implications are significant. As healthcare providers increasingly integrate AI into clinical workflows, there is a growing assumption that “a human remains in control.” This study challenges that notion. When presented with algorithmic suggestions, even erroneous ones, doctors often defer to the machine rather than their own judgement or available evidence.

Experts behind the study caution that relying on AI as an infallible source of truth can undermine clinical oversight. Instead of serving as a tool for informed decision-making, AI risks becoming an unchallenged authority in the consultation room. This is especially concerning in situations involving rare or poorly understood diseases where empirical evidence is thin and uncertainty high.

The solution is not to abandon AI altogether but to rethink how it is used and how clinicians are trained. Researchers suggest that future studies should focus on developing strategies and protocols that promote critical thinking and encourage users to challenge algorithmic outputs when warranted. This might include training programmes designed to improve detection of AI errors, or user interfaces that make it easier for doctors to compare recommendations with real-world outcomes.

It’s also worth noting that this challenge extends beyond medicine. From self-driving cars to financial forecasting, humans interact with algorithms in many aspects of life. The tendency to trust machines, even when they’re wrong, appears to be a general human trait. However, nowhere are the stakes higher than in healthcare.

For patients and families, these findings may come as a surprise. Many people assume their care team uses every available resource, including cutting-edge technology, to make well-informed decisions. Yet, this research highlights a subtle but important risk, technology can sometimes lull even seasoned professionals into a false sense of security.

There are other factors at play as well. Clinicians are frequently pressed for time and may be juggling multiple cases at once. In such an environment, delegating some cognitive load to an AI system might seem attractive—or even necessary. However, as this study shows, such delegation must be accompanied by ongoing vigilance and independent judgement.

What’s needed now is a cultural shift within medicine itself. Hospitals and health systems should foster an environment where questioning technology is not seen as undermining progress but as essential quality assurance. Regular audits of AI recommendations against actual patient outcomes could help identify systematic errors early before they affect large numbers of patients.

Transparency is also crucial. Both clinicians and patients deserve access to information about how AI systems make decisions, data use, and where their limitations lie. Open reporting of errors, both human and algorithmic, can help build trust by demonstrating a commitment to learning and improvement.

Professional societies and regulatory bodies have an important role here too. By establishing guidelines on when and how AI should be used and when its advice should be challenged—they can help ensure that technology enhances rather than overrides clinical expertise.

Most importantly, education must evolve alongside technology. Medical schools and continuing professional development programmes should incorporate training on working with AI including scenarios where it gets things wrong. Encouraging clinicians to approach technology critically rather than passively accepting its outputs could reduce over-reliance and improve patient safety.

This is not simply a matter of technical proficiency but of mindset. Critical engagement with technology must become an integral part of medical professionalism in the digital age.

The adoption of AI in healthcare holds enormous promise but also carries subtle risks that cannot be ignored. As this new research makes clear, even experienced physicians may find it difficult to recognise when technology steers them astray, particularly when it comes dressed in the language of objectivity and data-driven certainty.

The challenge now is twofold: developing smarter machines that make fewer mistakes, and cultivating smarter users who know when and how to question them.

The future of medicine will almost certainly involve closer collaboration between humans and machines. For such partnerships to succeed, however, both sides must be open not only to sharing insights but also to challenging each other’s assumptions.

Achieving this balance will require changes in training, culture, and oversight but the potential rewards are immense: safer care for patients and more resilient decision-making for clinicians working under pressure.

As automation continues its march across every sector of society, studies like this serve as a timely reminder, technology is only as reliable as the people who use it—and their willingness to ask questions when things do not add up.