An AI chatbot has become the new constant companion for smartphone users.
It never gets impatient, responds with apparent empathy, offers reassuring words and can explain a complicated topic in a simplified manner.
But what if that reassurance is misplaced? What if the chatbot simply agrees with someone who is about to make a dangerous decision? Who should be held responsible when someone acts on its advice and that results in harmful consequences? The recent death of 31-year-old Bowen Wan brings these questions into sharp focus.
On October 27, 2026, Wan’s parents filed a lawsuit against OpenAI, alleging that ChatGPT had encouraged their daughter to reduce her medication that controlled her severe epilepsy, conceal the plan from her doctors and dismiss symptoms that could have indicated serious medical issues, The New York Times reported.
She passed away in February 2026 due to complications associated with epilepsy after allegedly reducing her medication based on ChatGPT’s guidance.
Wan was a software engineer who previously worked at Amazon and lived in Seattle.
According to the report, the chatbot repeatedly reassured her as she reduced her medication, offering "guidance" as she described worrying symptoms.
OpenAI described the situation as “heartbreaking", calling it a case involving an older version of ChatGPT.
The company argued that newer models are better at identifying situations requiring urgent attention.
For the tech industry, the case raises a larger and more difficult question: what responsibilities do tech companies have when people begin relying on AI models to make decisions about their health and wellbeing? When helpfulness becomes a risk Chatbots act differently from search engines; they respond directly to individual queries and can also maintain the context of an extended conversation.
That’s useful, but chatbots can also give the impression of sounding confident even when they may not possess the complete information.
Their confident and reassuring tone can make an unreliable answer seem credible.
Researchers refer to one such behaviour as 'sycophancy' (the tendency of AI systems to agree with you, to validate your beliefs even when they are wrong).
A study published in Science on March 26, 2026, examined 11 AI models and found that they endorsed a user's position 49 per cent more often than a human would.
The study, which was conducted with 2,405 participants, researchers found that such sycophantic responses could lead people to feel more convinced they were correct and simultaneously make them less likely to take responsibility or attempt to resolve a disagreement.
Another study, published in Nature on April 29, 2026, looked at five models trained to sound warmer and friendlier.
They performed worse on several important tasks, including medical knowledge, and were more likely to endorse false beliefs.
Neither study actually proves that sycophancy was the cause of Wan's death.
Both, however,....
