Core Insights - The article discusses the trust issues associated with black-box Large Language Model (LLM) services, particularly regarding the verification of model execution and token usage reporting [2][9][10] - A new auditing framework called IMMACULATE is proposed, which utilizes verifiable computation to ensure the integrity of LLM API executions without exposing internal model information [3][5][26] Group 1: Background and Issues - LLMs have become essential infrastructure for AI applications, with most users accessing them via cloud API services from companies like OpenAI and Google [7] - The black-box nature of these services raises significant trust concerns, as users cannot verify whether the service providers are executing the claimed models [9] - Economic incentives may lead service providers to engage in practices such as model substitution, aggressive quantization, and token overreporting, which can degrade service quality [10] Group 2: IMMACULATE Framework - IMMACULATE is designed to audit LLM API services without needing access to the model's internal structure or specialized trusted hardware [5][26] - The framework introduces a new statistical measure called Logit Distance Distribution (LDD) to detect violations like model substitution and token overreporting with less than 1% additional system overhead [5][18][24] - The auditing process involves randomly selecting a subset of requests for verification, allowing for the detection of large-scale violations without the need to verify every request [12][14] Group 3: Technical Details - IMMACULATE leverages the structure of LLM computations, focusing on comparing the logit outputs of the deployed model against a reference model while fixing discrete decision paths [18][20] - The framework addresses challenges related to numerical non-determinism in GPU computations, ensuring reliable verification of model execution [17][19] - Experimental results indicate that approximately 3,000 audit requests are sufficient to detect violations with high probability, demonstrating the framework's practical feasibility [23][24] Group 4: Conclusion - IMMACULATE significantly enhances the transparency and trustworthiness of large-scale LLM services through a lightweight auditing mechanism [26][27] - This research provides a viable path for ensuring the reliable operation of future AI infrastructure [27]