Core Insights - The article discusses the limitations of large language models (LLMs) in performing analogy tasks, highlighting that while they may show some capability, they often rely on surface features rather than true abstract reasoning [1][2][29]. Group 1: Analogy Capabilities of Large Language Models - Research indicates that LLMs like GPT-3, GPT-3.5, and GPT-4 can perform analogy tasks but may not truly understand the underlying principles, as they often depend on memorized training data rather than genuine reasoning [2][3][4]. - A study found that when presented with altered analogy tasks using fictional or symbolic alphabets, the accuracy of LLMs significantly decreased, often falling below that of human participants, including children [7][10][13]. - The performance of LLMs in analogy tasks is sensitive to changes in the format of the questions, indicating a lack of robustness in their reasoning abilities [18][20][29]. Group 2: Comparison with Human Performance - In various analogy tasks, human participants, including children, consistently outperformed LLMs, especially when the tasks involved non-standard formats or required deeper understanding [10][13][24]. - The accuracy of LLMs like GPT-4 dropped significantly when faced with paraphrased stories or altered answer options, demonstrating their reliance on specific formats rather than flexible reasoning [25][26][29]. - The findings suggest that while LLMs can achieve high accuracy in controlled tasks, they struggle with variations that require a deeper understanding of context and relationships, unlike human reasoning [29][30]. Group 3: Implications for Future Development - The article emphasizes the need for further research to enhance the analogy-making capabilities of LLMs, suggesting that structured data from traditional literature could provide new avenues for improvement [30][31]. - It advocates for the development of robust testing frameworks to evaluate LLMs' adaptability to new situations, which is crucial for their application in critical fields such as education and healthcare [29][30]. - The potential for LLMs to generate new rules through analogy reasoning frameworks is highlighted as a promising direction for future advancements [30][34].