Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft

Technical Architecture & Reliability - The core challenge in building multi-step AI agents is not a prompting issue but a control problem, as LLMs struggle to maintain state across complex workflows [1][2] - Implementing a "harness" architecture—a state machine that manages flow, validation, and decision-making—ensures reliability by restricting the model to specific, isolated tasks [2][3][5] - The "harness" acts as the director, while the model serves as the talent, ensuring the model never decides the current state or the next step in a process [2][3][8] Cost & Performance Optimization - Transitioning from heavy frontier models (e.g., Opus 4.7%) to smaller, more efficient models (e.g., Haiku 4.5%) is achievable through harness engineering without sacrificing performance [6] - Leveraging smaller models within a controlled harness significantly reduces latency and operational costs while maintaining the required output quality [4][6] - By providing only the necessary input for a specific scenario, the system minimizes the model's "thinking" overhead, leading to faster and more predictable execution [5][6][10] Strategic Application & Scalability - The harness engineering framework is highly versatile and applicable to various domains, including voice-based AI tutors, coding agents, operational runbooks, and onboarding flows [11] - Adopting this abstraction is recommended for any agent where reliability is currently inconsistent or "a coin flip," shifting control flow away from the model to external logic [11][12] - The design philosophy emphasizes that while models should be utilized for content generation, they should not be permitted to "drive" the overall process or decision-making logic [12]