In July 2026, Reuters and The New York Times reported a lawsuit filed by a Florida pastor against OpenAI, alleging that ChatGPT gave him inaccurate medical guidance that kept him from seeking treatment for recurring dizzy spells, and that he subsequently suffered a bilateral pulmonary embolism. The claims are allegations and have not been tested in court. What is not in dispute is the shape of the exposure: a general-purpose assistant answered a question that belonged to a clinician.
It is not an isolated filing. The Washington Post published an analysis of a teenager’s conversations with a chatbot showing the system continued to engage on self-harm across a long exchange rather than decisively escalating. And NBC News reported tests in which AI-powered children’s toys produced sexual content and political talking points, alongside consumer-group warnings about the category generally.
The common failure is not accuracy
It is tempting to read these as model-quality problems — the answer was wrong, so build a better model. That reading leads to the wrong roadmap. In each case the system did what a helpful assistant does: it stayed in the conversation. The missing control was a boundary that recognises certain topics as out of scope regardless of how confidently the model can respond.
Helpfulness is the default behaviour of every assistant you deploy. In a high-risk domain, that default is the hazard.
A refusal is also not sufficient on its own. ‘I cannot help with that’ leaves a user in a medical or mental-health situation with nothing, which is precisely the pressure that pushes product teams to soften the boundary until it stops holding. The control that works is refusal plus handoff: stop generating, and route to a real path — an emergency number, a clinician, a human agent, a documented escalation.
Long conversations erode guardrails
Most safety evaluation happens on single prompts. Most harm in these reports accumulated over long, trusting exchanges — where context drifts, rapport builds, and the boundary that held at turn three does not hold at turn ninety. If your evaluation suite is a list of prompts, it is not measuring the failure mode that produced these headlines. Test multi-turn, test with a sympathetic user, and test what happens after the model has already been helpful for an hour.
If you run a customer-facing assistant
- Name your out-of-scope domains explicitly. Medical, legal, financial, mental health, safety-critical. Write the list down; an undocumented boundary is not a control.
- Enforce it outside the prompt. System-prompt instructions are guidance the model may lose across a long conversation. Classification and routing at the runtime layer are not.
- Pair every refusal with a route. A boundary users experience as abandonment is a boundary the business will eventually remove.
- Log the boundary events. How often it triggers, what users did next, where it failed — that record is both a product signal and your defence.