SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale - Rishi Desai, Abundant AI

Hi everyone. My name is Rishi Desai. I'm an ML engineer at Abundant AI, where we build reinforcement learning environments for Frontier Labs.Today, I'm going to talk about SWE Marathon, a benchmark that answers a question that is starting to matter a lot more. Can coding agents stay coherent over a billion token budget. Can they build Slack from scratch.Can they rewrite an entire JAX code base in PyTorch. Can they build a C compiler in Rust. This is what SWE Marathon is trying to measure.What happens when c ...