X @Avi Chawla

Key Concepts - KV caching accelerates inference by pre-computing the prompt's KV cache before token generation [1] - This pre-computation explains the longer time-to-first-token (TTFT) observed in models like ChatGPT [1] Performance Bottleneck - Time-to-first-token (TTFT) is a significant performance metric in inference [1] - Improving TTFT is an area for further research and development [1]