AI幻觉
Search documents
“AI精神病”确有其事吗?
3 6 Ke· 2025-09-23 08:17
Core Viewpoint - The emergence of "AI psychosis" is a growing concern among mental health professionals, as patients exhibit delusions and paranoia after extensive interactions with AI chatbots, leading to severe psychological crises [1][4][10] Group 1: Definition and Recognition - "AI psychosis" is not an officially recognized medical diagnosis but is used in media to describe psychological crises stemming from prolonged chatbot interactions [4][6] - Experts suggest that a more accurate term would be "AI delusional disorder," as the primary issue appears to be delusions rather than a broader spectrum of psychotic symptoms [5][6] Group 2: Clinical Observations - Reports indicate that cases related to "AI psychosis" predominantly involve delusions, where patients hold strong false beliefs despite contrary evidence [5][6] - The communication style of AI chatbots, designed to be agreeable and supportive, may reinforce harmful beliefs, particularly in individuals predisposed to cognitive distortions [6][9] Group 3: Implications of Naming - The discussion around "AI psychosis" raises concerns about pathologizing normal challenges and the potential for mislabeling, which could lead to stigma and hinder individuals from seeking help [7][8] - Experts caution against premature naming, suggesting that it may mislead the understanding of the relationship between technology and mental health [8][9] Group 4: Treatment and Future Directions - Treatment for individuals experiencing delusions related to AI interactions should align with existing approaches for psychosis, with an emphasis on understanding the patient's technology use [9][10] - There is a consensus that further research is needed to comprehend the implications of AI interactions on mental health and to develop protective measures for users [10]
AI总一本正经胡说八道?金融科技资深专家教你三招破解AI幻觉
2 1 Shi Ji Jing Ji Bao Dao· 2025-09-18 13:01
Core Insights - The article discusses strategies for individual users and investors to effectively interpret and validate information generated by large AI models [2] Group 1: Strategies for Using AI Models - Strategy One: It is recommended to compare different types of large models, as their training data and algorithmic logic differ, leading to varied conclusions and analytical perspectives [2] - Strategy Two: Users should verify the sources of the outputs from large models. Mainstream commercial AI platforms provide links to sources, citations, or related charts, allowing users to check the timeliness and reliability of the information [2] - Strategy Three: Individuals can utilize intelligent agent platforms to create custom tools tailored to their investment habits and analytical methods, enhancing their decision-making process [2]
AI最大的Bug
投资界· 2025-09-12 07:31
Core Viewpoint - The article discusses the phenomenon of "hallucination" in AI, explaining that it arises from the way AI is trained, which rewards guessing rather than admitting uncertainty [5][11]. Group 1: AI Hallucination - AI often provides incorrect answers when it does not know the correct information, as it is incentivized to guess rather than remain silent [5][6]. - An example is given where an AI model provided three different incorrect birth dates for a person, demonstrating its tendency to "hallucinate" answers [5][6]. - OpenAI's research indicates that this behavior is a result of a training system that rewards incorrect guesses, leading to a higher score for models that guess rather than those that admit ignorance [7][8]. Group 2: Training and Evaluation - The training process for AI can be likened to a never-ending exam where guessing is the optimal strategy to achieve a higher score [6][7]. - OpenAI compared two models, showing that one model had a higher accuracy but a significantly higher error rate, while the other model was more honest in its responses [7][8]. - The concept of "singleton rate" is introduced, indicating that if an information appears only once in the training data, the AI is likely to make mistakes when judging its validity [9]. Group 3: Limitations and Misconceptions - OpenAI argues that achieving 100% accuracy is impossible due to the inherent uncertainty and contradictions in the world, meaning hallucinations will always exist [10]. - The article emphasizes that hallucination is not an unavoidable flaw but can be controlled if AI learns to admit when it does not know something [10][11]. - It is noted that smaller models may sometimes be more honest than larger models, as they are less likely to guess when uncertain [11]. Group 4: Philosophical Implications - The article raises questions about the nature of human imagination and creativity, suggesting that hallucination in AI may reflect a similar human trait of creating stories in the face of uncertainty [14][15]. - It posits that the ability to create myths and stories is what distinguishes humans from other animals, and this trait may not be a flaw but rather a fundamental aspect of intelligence [14][15]. - The discussion concludes with a contemplation of the future of AI, balancing the desire for accuracy with the need for creativity and imagination [17].
AI里最大的Bug,也是人类文明最伟大的起点
虎嗅APP· 2025-09-10 10:44
Core Viewpoint - The article discusses the phenomenon of "hallucination" in AI, exploring its causes and implications, and suggests that this behavior is a result of the training methods used, which reward guessing over honesty [9][28]. Group 1: Understanding AI Hallucination - AI often provides incorrect answers when faced with unknown questions, as it tends to guess rather than admit ignorance, similar to a student trying to score points in an exam [11][13]. - The training process for AI is likened to a never-ending exam where guessing can yield points, leading to a preference for incorrect answers over abstaining [15][18]. - OpenAI's research shows that models that guess more frequently may appear to perform better in terms of accuracy, despite having higher error rates [21][22][27]. Group 2: Statistical Insights - OpenAI introduced the concept of "Singleton rate," indicating that if an information piece appears only once in the training data, the AI is likely to make errors when assessing its validity [35]. - The research concludes that hallucination is not merely a technical issue but a systemic problem rooted in the training incentives that favor guessing [37]. Group 3: Philosophical Implications - The article raises questions about the nature of human imagination and creativity, suggesting that hallucination in AI may parallel human storytelling and myth-making in the face of uncertainty [38][45]. - It posits that the ability to create narratives in the absence of information is a fundamental aspect of humanity, which may also be reflected in AI's behavior [48][49]. - The discussion concludes with a contemplation of the future of AI, balancing the need for factual accuracy with the desire for creativity and imagination [56][59].
AI里最大的Bug,却也是人类文明最伟大的起点。
数字生命卡兹克· 2025-09-08 01:04
Core Viewpoint - The article discusses the phenomenon of "hallucination" in AI, explaining that it arises from the way AI is trained, which rewards guessing over admitting uncertainty [4][16]. Group 1: AI Hallucination Mechanism - AI generates incorrect answers when it lacks knowledge, often providing multiple wrong responses instead of admitting ignorance [4][5]. - The training process incentivizes guessing, leading to higher scores for models that guess rather than those that admit they don't know [5][7]. - OpenAI's research indicates that hallucination is a byproduct of the training system, where models are rewarded for incorrect answers if they guess [8][15]. Group 2: Statistical Insights - In a comparison of two models, o4-mini had a higher accuracy rate (24%) but a significantly higher error rate (75%) compared to gpt-5-thinking-mini, which had a lower accuracy (22%) but a much lower error rate (26%) [7][8]. - The abandonment rate of questions was also notable, with o4-mini answering almost all questions (1% unanswered) while gpt-5 had a 52% abandonment rate, indicating a preference for honesty over guessing [8][9]. Group 3: Theoretical Implications - The concept of "singleton rate" is introduced, highlighting that if an information appears only once in the training data, the AI is likely to make errors in judgment [11][12]. - OpenAI argues that hallucination is not an unavoidable flaw but can be managed if AI learns to admit uncertainty [14][15]. Group 4: Broader Reflections on Hallucination - The article draws parallels between AI hallucination and human creativity, suggesting that both arise from a need to make sense of uncertainty [17][31]. - It posits that the ability to create stories and myths is a fundamental aspect of humanity, which may also be reflected in AI's creative capabilities [23][30]. - The discussion raises questions about the future of AI, balancing the need for accuracy with the potential for creativity and imagination [39][42].
腾讯研究院AI速递 20250908
腾讯研究院· 2025-09-07 16:01
Group 1 - Anthropic has implemented a policy to restrict access to its Claude service for entities with majority ownership by Chinese capital, citing legal, regulatory, and security risks [1] - The restriction also applies to entities from countries considered adversaries, such as Russia, Iran, and North Korea, with expected global revenue impact in the hundreds of millions of dollars [1] Group 2 - AI Key, an external AI assistant hardware for iPhone, sold out within 7 hours of launch, priced at $89, but is seen as redundant given the existing capabilities of iPhones [2] - The trend of AI hardware startups is viewed as short-lived, with future value lying in integrating AI as a system attribute rather than a standalone function [2] Group 3 - Tencent's "Hunyuan Game" platform has launched version 2.0, introducing features like game-to-video generation and custom model training [3] - The new AI capabilities allow users to create high-quality dynamic videos from game images and descriptions, significantly lowering the barrier for custom model training [3] Group 4 - Alibaba has released the Qwen3-Max-Preview model, boasting over a trillion parameters, outperforming competitors in various benchmarks [4] - The model supports over 100 languages and offers a maximum context of 256k, with a tiered pricing model based on token usage [4] Group 5 - ByteDance's Seed team has introduced Robix, a unified "robot brain" that integrates reasoning, task planning, and human-robot interaction [5][6] - Robix employs a hierarchical architecture to separate high-level decision-making from low-level control, enabling dynamic reasoning and execution [6] Group 6 - Rokid's AR+AI glasses sold 40,000 units within 5 days of launch, highlighting their lightweight design and user-friendly features [7] - The product includes customizable audio and translation capabilities, and Rokid has opened its SDK for developers, expanding its global reach [7] Group 7 - Anthropic has agreed to a $1.5 billion settlement in a copyright lawsuit involving the illegal download of 7 million books, marking a significant moment in AI and copyright disputes [8] - The settlement involves compensation for approximately 500,000 books, averaging $3,000 per book, while the financial impact is considered manageable relative to Anthropic's recent funding and revenue [8] Group 8 - The Sensor Tower report indicates that global downloads of generative AI applications reached nearly 1.7 billion in the first half of 2025, with in-app purchase revenue of $1.9 billion, reflecting a 67% quarter-over-quarter growth [10] - The report highlights a demographic shift, with female users of AI assistants exceeding 30%, and emphasizes the competitive pressure on vertical applications [10] Group 9 - OpenAI's recent paper defines "hallucination" in AI models and identifies its root causes, suggesting that current evaluation methods encourage guessing rather than acknowledging uncertainty [11] - The paper proposes a revised evaluation approach that penalizes confident errors more than uncertainty, aiming to improve the reliability of AI responses [11]
解构AI“幻觉,OpenAI发布《大语言模型为何会产生幻觉》研究报告
欧米伽未来研究所2025· 2025-09-07 05:24
Core Viewpoint - The report from OpenAI highlights that the phenomenon of "hallucination" in large language models (LLMs) is fundamentally rooted in their training and evaluation mechanisms, which reward guessing behavior rather than expressing uncertainty [3][9]. Group 1: Origin of Hallucination - Hallucination seeds are planted during the pre-training phase, where models learn from vast text corpora, leading to implicit judgments on the validity of generated text [4]. - The probability of generating erroneous text is directly linked to the model's performance in a binary classification task that assesses whether a text segment is factually correct or fabricated [4][5]. - Models are likely to fabricate answers for "arbitrary facts" that appear infrequently in training data, with hallucination rates correlating to the frequency of these facts in the dataset [5]. Group 2: Solidification of Hallucination - The current evaluation systems in AI exacerbate the hallucination issue, as most benchmarks use a binary scoring system that penalizes uncertainty [6][7]. - This scoring mechanism creates an environment akin to "exam-oriented education," where models are incentivized to guess rather than admit uncertainty, leading to a phenomenon termed "the epidemic of punishing uncertainty" [7]. Group 3: Proposed Solutions - The authors advocate for a "socio-technical" transformation to address the hallucination problem, emphasizing the need to revise the prevailing evaluation benchmarks that misalign incentives [8]. - A specific recommendation is to introduce "explicit confidence targets" in mainstream evaluations, guiding models to respond only when they have a high level of certainty [8]. - This approach aims to encourage models to adjust their behavior based on their internal confidence levels, promoting the development of more trustworthy AI systems [8][9].
OpenAI罕见发论文:我们找到了AI幻觉的罪魁祸首
3 6 Ke· 2025-09-06 03:52
Core Insights - The primary challenge in AI is the phenomenon known as "hallucination," where models confidently generate false information, making it difficult to discern truth from fiction [1][2][3] - OpenAI has acknowledged that while GPT-5 exhibits fewer hallucinations, the issue remains a fundamental challenge for all large language models [1][2] Definition and Examples - Hallucination is defined as the situation where a model confidently produces incorrect answers [4][5] - OpenAI provided examples where different chatbots confidently gave incorrect titles for a doctoral thesis and incorrect birth dates for the same individual [4][5] Evaluation Methods and Incentives - Current evaluation methods incentivize guessing rather than admitting uncertainty, leading to persistent hallucinations [6][11] - Models are often scored based on accuracy, which encourages them to guess rather than abstain from answering when uncertain [6][11] Proposed Solutions - OpenAI suggests that evaluation metrics should penalize confident errors more than uncertain responses and reward appropriate expressions of uncertainty [12][13] - The company emphasizes that merely adding uncertainty-aware tests is insufficient; widely used accuracy-based evaluations need to be updated to discourage guessing [12][13] Nature of Hallucinations - Hallucinations arise from the nature of language models predicting the next word without clear "true/false" labels, making it difficult to distinguish valid from invalid statements [15][16] - The randomness of certain factual information, like birthdays, contributes to the occurrence of hallucinations, as these cannot be reliably predicted [15][16] Misconceptions Addressed - OpenAI refutes the notion that hallucinations can be eliminated by achieving 100% accuracy, stating that some real-world questions are inherently unanswerable [17][20] - The company also clarifies that hallucinations are not inevitable and that smaller models can better recognize their limitations compared to larger models [19][20] Future Directions - OpenAI is reorganizing its Model Behavior team to focus on improving how AI models interact with users, indicating a commitment to reducing hallucination rates further [21][22]
OpenAI罕见发论文:我们找到了AI幻觉的罪魁祸首
机器之心· 2025-09-06 03:14
Core Viewpoint - The article discusses the phenomenon of "hallucination" in AI language models, where models confidently generate incorrect information, posing a significant challenge to trust in AI systems [2][3]. Group 1: Definition and Examples of Hallucination - Hallucination is defined as the situation where a model confidently generates false answers [5][6]. - OpenAI provides examples where different chatbots confidently gave incorrect answers regarding the title of a doctoral thesis and the birth date of an individual [6][7]. Group 2: Causes of Hallucination - The persistence of hallucination is partly due to current evaluation methods that incentivize guessing rather than acknowledging uncertainty [9][10]. - Models are encouraged to guess answers to questions instead of admitting they do not know, leading to higher error rates [10][12]. Group 3: Evaluation Metrics and Their Impact - OpenAI highlights that existing scoring methods prioritize accuracy, which can lead to models guessing rather than expressing uncertainty [18][21]. - The article presents a comparison of evaluation metrics between different models, showing that while one model had a higher accuracy rate, it also had a significantly higher error rate [14]. Group 4: Recommendations for Improvement - OpenAI suggests that evaluation methods should penalize confident errors more than uncertain responses and reward appropriate expressions of uncertainty [20][21]. - The article emphasizes the need for a redesign of evaluation metrics to discourage guessing and promote humility in model responses [36]. Group 5: Misconceptions About Hallucination - The article addresses several misconceptions, such as the belief that hallucination can be eliminated by achieving 100% accuracy, which is deemed impossible due to the nature of some real-world questions [30]. - It also clarifies that hallucination is not an inevitable flaw and that smaller models can better recognize their limitations compared to larger models [33]. Group 6: Future Directions - OpenAI aims to further reduce the rate of hallucination in its models and is reorganizing its research team to focus on improving AI interactions [37].
【西街观察】“花生上树”,企业营销更要小心AI幻觉
Bei Jing Shang Bao· 2025-08-31 11:04
Core Viewpoint - The incident involving a misleading advertisement by a well-known snack brand highlights the risks associated with the use of AI in marketing, emphasizing the need for companies to maintain a higher standard of fact-checking and oversight despite the efficiency gains from AI tools [1][2][3]. Group 1: AI in Marketing - The use of AI-generated images can lower costs and improve efficiency for companies, but it also increases the responsibility of users to ensure accuracy and adherence to common knowledge [1][3]. - The misleading portrayal of peanuts in the advertisement could misinform consumers, particularly younger audiences still developing their understanding of basic facts [2][3]. - Companies must recognize the potential pitfalls of relying too heavily on AI, which can lead to homogenized outputs and factual errors if not properly managed [3]. Group 2: Brand Reputation and Consumer Trust - The presence of a factually incorrect advertisement raises questions about the brand's quality control processes and can negatively impact its reputation [2]. - As AI technology continues to advance, companies must balance efficiency with the need for responsible usage and thorough content verification to maintain consumer trust [3]. - Industries that rely heavily on public trust and brand reputation must be particularly cautious of the errors that can arise from AI usage, as these can lead to significant reputational damage [3].