Core Viewpoint - The article discusses how ZTE's AIM team has developed two new evaluation metrics and a repository-level code corpus processing framework to enhance AI code completion tools, making them more aligned with developer needs [1][2]. Group 1: New Evaluation Metrics - The team introduced two new metrics: Longest Common Prefix (LCP) and ROUGE-LCP, which are designed to better reflect user perceptions of code completion quality [6][8]. - LCP focuses on the longest continuous matching characters from the start of the output sequence, emphasizing the importance of the initial part of the AI's suggestion for user acceptance [10]. - ROUGE-LCP normalizes LCP by the length of the reference sequence, allowing for fair comparisons across different lengths of completion samples [12]. Group 2: Code Corpus Processing Framework - The SPSR-Graph framework was developed to help AI models understand complex code repository structures and semantic dependencies, moving beyond limited contextual understanding [14][15]. - This framework constructs a specialized code knowledge graph that models structural information and cross-file dependencies, enhancing the depth of understanding for the AI model [15][19]. - The process includes strict data filtering, AST-based semantic unit extraction, and the construction of a directed graph to represent dependencies among code units [20][30]. Group 3: Experimental Results - The team conducted experiments to validate the effectiveness of the new metrics and methods, analyzing over 10,000 real user data records from ZTE-Code-Copilot [27]. - A significant positive correlation was found between LCP values and user acceptance rates, with Pearson correlation coefficients exceeding 0.69, indicating that higher LCP values lead to increased user adoption [31][38]. - The new metrics outperformed traditional evaluation metrics in correlating with user acceptance rates, demonstrating their ability to capture user behavior and intent more accurately [43]. Group 4: Future Prospects - The team aims to further explore the adaptability of LCP and ROUGE-LCP metrics across various code generation tasks and model types [51]. - There are plans to integrate the SPSR-Graph method with reinforcement learning techniques to enhance the model's reasoning capabilities and expand its application to more complex software engineering domains [52].