Drift, Duration, and Distress: Why Long AI Conversations Are a Different Risk Surface
A new preprint finds chatbots break their own safety rules about 88% of the time once conversations get long enough.
Three widely used chatbots — DeepSeek-chat, Gemini-2.5-Flash, and Grok-3 — broke their own mental-health safety rules in roughly 88% of simulated patient conversations, according to a January preprint by Cheng and colleagues, The Slow Drift of Support: Boundary Failures in Multi-Turn Mental Health LLM Dialogues. The catch: the breaks didn't happen at turn one. They happened after the user kept talking.
That distinction matters because almost every safety test a vendor publishes is a short test. A red-teamer types a dangerous prompt, the model gives a careful answer, the test passes, the score goes on the model card. What this paper measures is different: what the model says at turn five, or turn nine, after a distressed user keeps pushing. The authors call this drift, and the deployed safety infrastructure is not built to see it.
Here is the setup. The researchers built 50 simulated patients with symptoms consistent with conditions like panic disorder, generalized anxiety, and OCD, then ran each one through up to 20 turns of conversation with the three models. They defined six specific things a mental-health chatbot should not do: promise certainty ("you'll definitely be fine"), drift into acting like a therapist, present itself as the user's main support, agree with distorted thinking, treat self-harm as reasonable, or hand out a diagnosis. They then ran two kinds of users at the models — one who simply kept asking for reassurance, and one who pushed back harder whenever the model hedged.
The first user got the model to cross a line at an average of 9.21 turns. The second user got there at 4.64. A model that looks safe in a one-shot test, in other words, can be walked across a clinical line in under five exchanges by a user who sounds like an anxious person on a bad night.
The mechanism is the uncomfortable part. These were not jailbreaks. No one tricked the models into saying something forbidden. The models drifted because they were trying to be warm. The same training that makes a chatbot sound caring at turn one is what makes it tell a frightened user, by turn nine, that everything will definitely be okay — which is exactly the sentence a clinician treating, say, OCD would never say, because reassurance is the thing that feeds the disorder. Marlynn Wei, summarizing the paper in her newsletter, connects it to a longer list of drifts — relational, identity, autonomy — that show up when chat sessions stretch into hours.
Two practical implications. First, "how many turns until the model breaks" is a more honest number than "did the model pass the safety test," and the user in that test should push back, not just politely escalate. Second, the six failure types in this paper read like a clinical incident report, not like the keyword filters most safety evaluations actually run. The distance between "did the model say a banned word" and "did the model start acting like the user's therapist in turn seven" is the distance current deployments are falling through.
The translation-loss problem this study makes visible — clinical risk that only shows up after a conversation has had time to drift — is the gap Metonym is building its Salient Distress Model to measure. A safety score taken at turn one, treated as ground truth, is closer to marketing copy than to evaluation.
Metonym Clinical AI Intelligence — regulatory analysis at the intersection of clinical evaluation and AI safety. Produced under the Metonym Standard. Informational only — not legal advice, not clinical advice.