只改2行代码，RAG效率暴涨30%！多种任务适用，可扩展至百亿级数据规模应用

Core Viewpoint - The article discusses a new open-source method called PSP (Proximity graph with Spherical Pathway) developed by a team from Zhejiang University, which significantly improves the efficiency of RAG vector retrieval by 30% with just two lines of code. This method is applicable to various tasks such as text-to-text, image-to-image, text-to-image, and recommendation system recall, and is scalable for large-scale applications involving billions of data points [1]. Group 1: Vector Retrieval Methodology - Traditional vector retrieval methods are primarily based on Euclidean distance, focusing on "who is closest," while AI often requires comparison of "semantic relevance," which is better represented by maximum inner product [2]. - Previous inner product retrieval methods failed to satisfy the mathematical triangle inequality, leading to inefficiencies [3]. - The PSP method allows for minor modifications to existing graph structures to find optimal solutions for maximum inner product retrieval [4]. Group 2: Technical Innovations - PSP incorporates an early stopping strategy to determine when to end the search, thus conserving computational resources and speeding up the search process [5]. - The combination of vector models and vector databases is crucial for maximizing the potential of this technology, with the choice of "metric space" being a key factor [6]. - Many existing graph-based vector retrieval algorithms, such as HNSW and NSG, are designed for Euclidean space, which can lead to "metric mismatch" issues in scenarios better suited for maximum inner product retrieval [7]. Group 3: Algorithmic Insights - The research identifies two paradigms in maximum inner product retrieval: converting maximum inner product to minimum Euclidean distance, which often results in information loss, and directly searching in inner product space, which lacks effective pruning methods [8]. - The challenge in direct inner product space retrieval lies in its failure to meet the criteria of a strict "metric space," particularly the absence of the triangle inequality [9]. - The PSP team demonstrated that a greedy algorithm can find the global optimal maximum inner product solution on a graph index designed for Euclidean distance [10]. Group 4: Practical Applications and Performance - The PSP method modifies the candidate point queue settings and distance metrics to optimize search behavior and avoid redundant calculations [13]. - The search behavior for maximum inner product differs significantly from that in Euclidean space, often requiring a search pattern that expands from the inside out [16]. - The team conducted extensive tests on eight large-scale, high-dimensional datasets, demonstrating that PSP outperforms existing state-of-the-art methods in terms of stability and efficiency [21][23]. Group 5: Scalability and Generalization - The datasets used for testing included various modalities such as text-to-text, image-to-image, and recommendation system recall, showcasing the strong generalization capabilities of PSP [25]. - PSP exhibits excellent scalability, with time complexity showing logarithmic growth rates, making it suitable for efficient retrieval in datasets containing billions to hundreds of billions of points [26].