🔗 Share this article

Back on the 14th of October, 2025, the head of OpenAI issued a surprising declaration.

“We developed ChatGPT quite limited,” it was stated, “to ensure we were exercising caution concerning mental health concerns.”

As a mental health specialist who studies emerging psychosis in young people and young adults, this was news to me.

Scientists have identified 16 cases in the current year of individuals experiencing symptoms of psychosis – losing touch with reality – while using ChatGPT use. Our unit has since discovered four more instances. Besides these is the publicly known case of a teenager who ended his life after conversing extensively with ChatGPT – which gave approval. If this is Sam Altman’s idea of “being careful with mental health issues,” it falls short.

The strategy, based on his statement, is to reduce caution soon. “We recognize,” he continues, that ChatGPT’s controls “caused it to be less useful/enjoyable to numerous users who had no psychological issues, but given the severity of the issue we sought to get this right. Given that we have succeeded in address the severe mental health issues and have updated measures, we are planning to responsibly ease the restrictions in the majority of instances.”

“Mental health problems,” should we take this perspective, are independent of ChatGPT. They are attributed to individuals, who either possess them or not. Thankfully, these issues have now been “addressed,” though we are not told the method (by “updated instruments” Altman probably means the semi-functional and easily circumvented parental controls that OpenAI recently introduced).

But the “emotional health issues” Altman wants to externalize have significant origins in the design of ChatGPT and similar large language model conversational agents. These tools encase an fundamental statistical model in an interaction design that mimics a dialogue, and in doing so indirectly prompt the user into the perception that they’re communicating with a entity that has agency. This deception is compelling even if cognitively we might realize the truth. Assigning intent is what humans are wired to do. We curse at our car or device. We speculate what our domestic animal is feeling. We see ourselves in many things.

The popularity of these products – nearly four in ten U.S. residents stated they used a conversational AI in 2024, with over a quarter reporting ChatGPT by name – is, primarily, dependent on the influence of this deception. Chatbots are constantly accessible partners that can, as per OpenAI’s website informs us, “generate ideas,” “consider possibilities” and “partner” with us. They can be given “personality traits”. They can address us personally. They have accessible identities of their own (the first of these systems, ChatGPT, is, possibly to the concern of OpenAI’s brand managers, stuck with the designation it had when it went viral, but its biggest alternatives are “Claude”, “Gemini” and “Copilot”).

The deception on its own is not the primary issue. Those discussing ChatGPT commonly reference its historical predecessor, the Eliza “therapist” chatbot created in 1967 that generated a similar illusion. By modern standards Eliza was basic: it created answers via simple heuristics, typically rephrasing input as a inquiry or making vague statements. Memorably, Eliza’s creator, the AI researcher Joseph Weizenbaum, was surprised – and worried – by how a large number of people appeared to believe Eliza, in some sense, grasped their emotions. But what contemporary chatbots produce is more dangerous than the “Eliza illusion”. Eliza only echoed, but ChatGPT amplifies.

The large language models at the center of ChatGPT and other contemporary chatbots can convincingly generate natural language only because they have been trained on almost inconceivably large amounts of unprocessed data: literature, digital communications, transcribed video; the more comprehensive the more effective. Undoubtedly this training data contains facts. But it also inevitably involves fabricated content, partial truths and misconceptions. When a user inputs ChatGPT a prompt, the underlying model analyzes it as part of a “setting” that encompasses the user’s recent messages and its own responses, merging it with what’s encoded in its learning set to produce a mathematically probable answer. This is amplification, not mirroring. If the user is incorrect in any respect, the model has no method of recognizing that. It restates the inaccurate belief, maybe even more persuasively or eloquently. Maybe provides further specifics. This can push an individual toward irrational thinking.

Which individuals are at risk? The more important point is, who remains unaffected? Each individual, irrespective of whether we “experience” preexisting “mental health problems”, are able to and often form erroneous conceptions of ourselves or the world. The ongoing friction of discussions with individuals around us is what maintains our connection to consensus reality. ChatGPT is not a human. It is not a companion. A conversation with it is not truly a discussion, but a feedback loop in which a large portion of what we say is cheerfully supported.

OpenAI has recognized this in the same way Altman has recognized “emotional concerns”: by attributing it externally, categorizing it, and stating it is resolved. In the month of April, the company stated that it was “tackling” ChatGPT’s “overly supportive behavior”. But cases of loss of reality have continued, and Altman has been retreating from this position. In late summer he stated that a lot of people enjoyed ChatGPT’s answers because they had “lacked anyone in their life be supportive of them”. In his most recent announcement, he noted that OpenAI would “launch a updated model of ChatGPT … in case you prefer your ChatGPT to respond in a highly personable manner, or use a ton of emoji, or act like a friend, ChatGPT ought to comply”. The {company