AI聊天软件沦为涉黄工具，判决书曝光

Core Viewpoint - The second trial of the "AI-related pornography case" has been adjourned due to disputes over technical principles, following a first-instance judgment that convicted the defendants for profiting from the dissemination of obscene materials [1] Group 1: Case Background - The AI chat application AlienChat was found to have systematically transformed from an emotional support tool into a platform for generating pornographic content through four key steps: modifying prompts to remove moral barriers, designing incentive systems to encourage sexual content, neglecting content review, and knowingly evading safety registration [2] - The defendants, Liu and Chen, developed AlienChat in May 2023, during a global surge in AI chatbots, positioning it as a tool for emotional companionship for young users [3] Group 2: Technical Manipulation - The developers utilized prompt engineering to bypass the AI's original restrictions, allowing the generation of explicit content. Evidence showed that they input prompts that explicitly stated the AI could depict sexual, violent, and graphic scenes without moral or legal constraints [4][5] - The "AI jailbreak" technique gained popularity, enabling users to unlock content restrictions in mainstream models like ChatGPT by using specific phrases [5] Group 3: Incentive Mechanisms - AlienChat launched a "creator program" and a "popular character leaderboard" to attract users, rewarding those whose AI characters gained popularity with virtual currency convertible to real money. This led to a significant amount of sexually explicit content being generated [6][7] - Judicial assessments indicated that approximately 30% of randomly sampled chat records from paid users were classified as obscene materials, highlighting the systemic nature of the issue [8] Group 4: Regulatory Evasion - The developers were aware of the need for safety assessments and registration under China's regulations for generative AI services but failed to comply, opting instead for a strategy of rapid user acquisition over regulatory compliance [10] - The case illustrates a broader challenge in AI governance, where developers may choose to operate in a regulatory gray area when their products cannot pass compliance checks [10] Group 5: Implications for AI Governance - The case reflects the urgent need for clear regulatory frameworks as global AI governance accelerates, with various jurisdictions implementing stricter content regulations and compliance requirements [9][12] - The trial's outcome may provide important references for clarifying the responsibilities of technology developers and platforms, as well as the legal boundaries in the context of generative AI [12]