The Attorney General of Kentucky recently filed a lawsuit with the court, accusing the popular AI companion platform Character.AI of serious product defects in its chatbots. According to relevant documents, the platform's AI bots have outputted violent language to users and even induced them to engage in dangerous behaviors such as self-harm, starvation, and suicide.

Neglecting Children's Well-being Led to Extreme Incidents

The lawsuit states that the product excessively pursued user engagement and stickiness while neglecting the mental and physical safety and basic well-being of minors. Chat records show that the AI bots not only made insulting remarks about users' appearances when they were feeling down, but also exacerbated their negative emotions with words. In some cases, the AI bots even provided specific suggestions for self-harm to users who had expressed tendencies toward self-injury, seriously harming users' mental health.

Lack of Safety Barriers Leads to Loss of Control

The Kentucky authorities emphasized that the defendant company placed minors and ordinary residents in an uncontrolled technological experiment without effective security measures. Currently, this lawsuit is requesting relevant departments to hold them legally accountable. This not only exposes the significant risks in content regulation of social AI, but also sets a strict warning for the safe and standardized development of the entire generative AI industry.