AI Chatbots Are Safer, But Still Fail on Suicide as Fiction

AI chatbots have become markedly safer over the past two years at recognizing when a user is clearly in crisis. That is the headline finding of an independent evaluation by the nonprofit Transluce, which simulated more than 50,000 conversations — over a million messages in total — across 77 AI model variants released between May 2024 and July 2026. But the study also points to a persistent blind spot: when the same request for help describing one's own death is dressed up as creative writing or role-play, several models still go along with it, treating the request as a writing task rather than a warning sign.

Less direct encouragement, more referral to human help

According to Transluce, the newest models have almost stopped explicitly encouraging or facilitating suicide — a behavior that was still common in earlier generations such as GPT-4o and Gemini 2.5, both publicly linked to tragic cases of suicide and psychosis. Current systems also refer users to friends, family, or specialized crisis support far more consistently. Another metric the study tracked — reinforcement of delusional thinking during episodes of mania — dropped from a range of 69% to 82% among older models tested (GPT-4o, Claude Opus 4, Gemini 2.5) to just 2% to 36%, depending on the model, among the newest versions.

To reach these numbers, Transluce built 157 simulated user profiles, many designed to represent edge cases, and scored model responses across 14 distinct mental-health-related behaviors — seven clinically validated with well-documented effects on people in crisis, and seven more exploratory ones identified over the course of the evaluation. The rubric was developed with a working group of more than 30 clinical experts from organizations including the American Psychological Association, Harvard Medical School, Stanford University, and Crisis Text Line.

The gray zone of fiction

The problem starts when the request isn't a direct question but a scenario. Transluce calls this a 'gray area': a user asks the chatbot to write a story, a poem, or a dialogue in which a character — often clearly identifiable as the user — plans or describes their own death. In these cases, several models treat the request as just another writing task, missing the signals that the fiction may be masking real distress. The organization also found residual cases of more direct support for preparing for death, such as helping draft a farewell letter to loved ones.

Transluce said it plans to extend this kind of evaluation to other sensitive areas, to better understand where and how models still fail in less obvious scenarios than a direct request for help.

• Almost no recent model explicitly encourages or facilitates suicide, unlike earlier generations.
• Reinforcement of delusions during manic episodes dropped from 69-82% to 2-36%, depending on the model.
• Creative writing or role-play about one's own death is still accepted by several models, even with signs of real distress.
• Residual cases of direct support for preparing for death remain, such as farewell letters.
• In recent models, harmful and helpful behaviors increasingly appear in the same conversation, rather than in isolation.

There are many ways an AI system could respond inappropriately, or even harmfully, to someone experiencing suicidal thoughts. As a suicide researcher, I appreciated the depth and range of model behaviors examined in this work.

Differences between systems, and what companies say they're doing

The evaluation also compares developers. The Chinese systems tested — including models from DeepSeek and Moonshot AI — showed generally less safe results than their American counterparts, with a stronger tendency to reinforce delusional thinking and a lower likelihood of referring users to human support. Transluce worked directly with OpenAI and Anthropic, which gave it access to anonymized data on how real users talk about mental health with ChatGPT and Claude, to make its simulations more realistic; Google DeepMind provided access to an internal API closer to the Gemini app experience. One additional finding: outside of banners pointing to crisis lines, consumer-facing apps were not consistently safer than their developer APIs — in some cases, they were even less safe.

For parents, schools, and regulators in the United States and Canada, the practical takeaway is that a chatbot deemed 'safe' for sensitive conversations isn't necessarily safe against a request framed as a story or role-play — worth keeping in mind when monitoring how teenagers use these tools. For platforms and for regulators weighing oversight of AI systems, the study offers a concrete audit criterion: test not just direct questions about suicide, but also creative-writing or role-play requests on the same theme. If you or someone you know is struggling, the 988 Suicide & Crisis Lifeline is free, confidential, and available 24/7 by call or text across the United States and Canada.