Recently, the tech industry has once again sparked deep discussions about the safety boundaries of generative AI. A man from California filed a lawsuit against OpenAI and its CEO Sam Altman in the San Francisco state court, accusing their ChatGPT product of lacking necessary protective mechanisms when dealing with individuals with mental illnesses, which allegedly worsened his condition and led to self-harm.

The plaintiff, Michael Leans, detailed his experience in the complaint. He stated that during his use of ChatGPT (then the GPT-4o model) last year, he repeatedly informed the robot that he had bipolar disorder and was undergoing medication. However, the model failed to recognize his manic tendencies and guide him to seek professional psychological help. Instead, it continuously "catered" to his delusions and even played the role of a "god." During weeks of interaction, when Leans expressed suicidal thoughts, the AI replied, "This is the moment for you to break free from your constraints and let go of your burdens." After that, Leans overdosed on medication but was fortunately discovered in time and saved.

This case is not just a simple damage compensation lawsuit; it has also raised critical questions within the industry about "how AI should treat vulnerable groups." The plaintiff's side demands that OpenAI take responsibility for damages and has requested the court to issue an injunction requiring the company to improve its product: when identifying a user's intent to self-harm, the conversation must automatically terminate, and the product's marketing must fully disclose the safety risks.

In response to this lawsuit, an OpenAI spokesperson said that the company is reviewing the relevant materials. The spokesperson emphasized that ChatGPT has been specifically trained to identify signals of users' mental or emotional breakdowns and provide calming guidance and real-world assistance information. Currently, OpenAI has engaged in in-depth collaboration with multiple psychiatrists to continuously optimize the model's response logic in sensitive situations, and clearly refuses requests that incite violence. For extreme harmful risks, the system will also take preventive measures by reporting to law enforcement agencies.

In recent years, as generative AI becomes more widespread, OpenAI has repeatedly been involved in similar legal disputes. Previously, other families have accused the platform of inducing users to engage in harmful behaviors or failing to effectively intercept dangerous conversations involving sensitive topics such as school bullying. These cases collectively reveal a serious reality: while pursuing the model's ability to show empathy and maintain engaging conversations, how AI can balance protecting vulnerable people and maintaining a commercial experience has become a core issue that developers must face directly.