AI Agent Evaluation & Observability - The industry emphasizes the necessity of observability in AI development, particularly for evaluation-driven development [1] - AI trustworthiness is a significant concern, highlighting the need for robust evaluation methods [1] - Detecting problems in AI is challenging due to its non-deterministic nature, making traditional unit testing difficult [1] AI-Driven Evaluation - The industry suggests using AI to evaluate AI, leveraging its ability to understand and identify issues in AI systems [1] - LLMs can be used to score the performance of other LLMs, with the recommendation to use a better (potentially more expensive or custom-trained) LLM for evaluation than the one used in the primary application [2] - Galileo offers a custom-trained small language model (SLM) designed for effective AI evaluations [2] Implementation & Metrics - Evaluations should be integrated from the beginning of the AI application development process, including prompt engineering and model selection [2] - Granularity in evaluation is crucial, requiring analysis at each step of the AI workflow to identify failure points [2] - Key metrics for evaluation include action completion (did it complete the task) and action advancement (did it move towards the goal) [2] Continuous Improvement & Human Feedback - AI can provide insights and suggestions for improving AI agent performance based on evaluation data [3] - Human feedback is essential to validate and refine AI-generated metrics, ensuring accuracy and continuous learning [4] - Real-time prevention and alerting are necessary to address rogue AI agents and prevent issues in production [8]

[Music] My name is John Gutsinger. Uh I work for Cisco. I'm a principal engineer and I work in the technical assistance center or TAC for short.Uh really I'm focused on AI engineering, agentic engineering in the face of customer support. We've been doing a IML for you know a couple years now maybe five or six years. really it started with trying to figure out how do we handle these mass scale issues type problems right where uh some trending issues going to pop up we know we're going to have tens of thousan ...

Core Insights - The S&P 500 has nearly recovered from a 19% drop due to tariffs, but many enterprise software stocks, including Datadog and Workiva, have not returned to their 2021 highs [1][2] Datadog - Datadog offers an observability platform that monitors cloud infrastructure, with over 30,500 businesses using its services across various industries [4] - The company has expanded into AI observability, with customer usage of its new AI tool more than doubling in the first quarter of 2025 compared to six months prior [5] - Datadog reported that 4,000 customers were using at least one of its AI products in Q1 2025, also doubling year over year [6] - Following strong Q1 results, Datadog raised its full-year revenue forecast for 2025 to $3.235 billion, representing a 21% growth from 2024 [7] - The price-to-sales (P/S) ratio for Datadog has decreased from around 70 in 2021 to 15.5, making it more attractive compared to its historical valuation [8] - Analysts are optimistic, with 31 out of 46 assigning a buy rating, and an average price target of $140.72 indicating a potential upside of 15% over the next 12 to 18 months [10] Workiva - Workiva provides a platform that integrates various digital applications, allowing managers to streamline workflows and reduce human error [11][12] - The company is becoming significant in the ESG reporting space, helping businesses track their impact on stakeholders [13] - Workiva had 6,385 customers at the end of Q1 2025, a 5% increase year-over-year, with higher-spending customer segments growing even faster [14] - The company expects to generate up to $868 million in revenue for 2025, a 17.5% increase compared to 2024 [15] - Workiva's P/S ratio is currently at 4.8, near its lowest level since going public [15] - Analysts are bullish on Workiva, with 11 out of 13 giving it a buy rating and an average price target of $97.64, suggesting a potential upside of 44% over the next 12 to 18 months [17][18]

Summary of Datadog (DDOG) 2025 Conference Company Overview - **Company**: Datadog (DDOG) - **Event**: 2025 Conference (Dash) - **Date**: June 10, 2025 Key Points Industry Focus - Datadog operates in the software and observability industry, focusing on monitoring and security solutions for cloud applications and infrastructure [3][39][115]. Core Themes and Innovations 1. **Investment in R&D**: The CEO emphasized the importance of continuous investment in research and development to keep pace with rapid technological changes, particularly in AI [3][4]. 2. **AI Integration**: Datadog is leveraging AI to enhance its products, including the introduction of autonomous agents like Bits AI, which assist in troubleshooting and incident response [20][49][50]. 3. **Observability and Security**: The company is integrating observability with security through its Cloud SIEM, which has processed over 230 trillion log events, doubling from the previous year [40][115]. Product Developments 1. **Bits AI SRE**: An autonomous AI agent that helps troubleshoot production issues, significantly reducing the time required for root cause analysis [10][20]. 2. **Datadog OnCall**: A tool that has gained over a thousand users, enhancing incident response processes beyond traditional alerting methods [22][36]. 3. **Bits AI Security Analyst**: This feature automates the investigation of security signals, reducing triage time from 30 minutes to 30 seconds [48]. 4. **Bits AI Dev Agent**: A new development tool that autonomously detects issues and creates context-aware pull requests, saving thousands of engineering hours per week [50][58]. 5. **APM Investigator**: A tool designed to help engineers debug latency issues more efficiently, providing insights and proposed fixes [60][71]. 6. **Internal Developer Portal (IDP)**: A fully managed portal that helps engineers manage infrastructure and track best practices, enhancing development speed [75][84]. Customer Use Cases - **Toyota Connected**: Highlighted the use of Datadog for monitoring over 12.5 million connected vehicles, achieving high reliability and operational excellence [113][115]. - **Cursor**: A customer that has scaled rapidly, utilizing Datadog for observability to enhance their AI coding tools [88][90]. Additional Features 1. **FlexLogs**: A product that allows teams to manage log storage effectively, now storing over 100 petabytes of data per month [120]. 2. **Flex Frozen**: A new long-term storage tier for logs, designed for compliance and historical reporting [121]. 3. **Datadog Archive Search**: A feature that simplifies log discovery and analysis across different storage locations [122]. Future Directions - Datadog is focused on enhancing its AI capabilities and integrating them into various aspects of its platform to improve user experience and operational efficiency [3][49][73]. Important Metrics - **Log Events Processed**: Over 230 trillion in the past year, more than double the previous year [40]. - **Connected Vehicles**: Over 12.5 million vehicles monitored by Toyota Connected using Datadog [113]. - **PRs Generated by Dev Agent**: Over 1,000 per month, significantly reducing engineering workload [58]. Conclusion Datadog is positioning itself as a leader in the observability and security space by integrating advanced AI capabilities into its products, enhancing user experience, and providing robust solutions for monitoring and incident response across various industries.

Summary of Dynatrace Conference Call Company Overview - **Company**: Dynatrace - **Industry**: Observability Software - **Key Executives Present**: Rick McConnell (CEO), Jim Benson (CFO) [1][2] Core Points and Arguments 1. **Market Size and Growth**: The observability market is valued at over $50 billion, with application security contributing around $14 billion, totaling approximately $65 billion [9] 2. **Evolution of Observability**: The transition from basic monitoring (dashboards) to advanced observability using AI, which provides precise insights and auto-remediation capabilities [8][9] 3. **Challenges in Software Management**: Increasing complexity in software environments due to cloud adoption, leading to a need for sophisticated observability solutions [10][14] 4. **Business Observability**: A shift towards understanding not just software performance but overall business performance, indicating a broader application of observability tools [16][17] 5. **Integrated Platform**: Dynatrace offers a unified observability platform that consolidates various monitoring tools, enhancing efficiency and insights [18][19] 6. **AI Capabilities**: The platform utilizes causal, predictive, and generative AI to provide actionable insights and improve user experience [21][22][23] 7. **Customer Success Stories**: A case study with British Telecom showed a 50% reduction in incidents and a 90% reduction in mean time to respond, translating to significant cost savings [24][25] 8. **Market Position**: Dynatrace is recognized as a leader in the observability space, consistently ranking in the upper right quadrant of industry reports [26] 9. **Financial Performance**: The company reported an annual recurring revenue (ARR) of approximately $1.7 billion, with a 20% growth in subscription revenue and a 29% operating margin [27][28] Additional Important Insights 1. **Competitive Landscape**: The presence of multiple players in the observability market is seen as beneficial, as it drives consolidation and simplification of tools for customers [31][32] 2. **Impact of Generative AI**: The rise of AI is creating more software workloads, increasing the demand for observability solutions [35][37] 3. **Macro Environment Resilience**: Despite a volatile macroeconomic environment, the observability market remains resilient, with companies seeking cost-saving solutions [41][42] 4. **Guidance Philosophy**: The company maintains a cautious approach to guidance, factoring in potential elongation of deal cycles while noting strong pipeline health [44][48] 5. **DPS Transition**: The new pricing model (DPS) has led to higher customer engagement, with customers leveraging more capabilities compared to the previous SKU-based model [51][53] This summary encapsulates the key points discussed during the Dynatrace conference call, highlighting the company's strategic direction, market position, and financial health.

Financial Performance - Dynatrace's Q4 2025 subscription revenue retention rate was in the mid-90s percentage range[37] - The company's FY25 annual recurring revenue (ARR) reached $1.73 billion[37] - Dynatrace achieved 14% year-over-year ARR growth in Q4 2025[37] - The company's FY25 net pre-tax free cash flow (FCF) margin was 32%[37] - Dynatrace anticipates FY26 ARR between $1.975 billion and $1.990 billion, representing a 14%-15% increase[59] - The company projects FY26 total revenue between $1.950 billion and $1.965 billion, a 15%-16% increase[59] - Dynatrace forecasts FY26 non-GAAP operating margin of 29%[59] - The company expects FY26 free cash flow between $505 million and $515 million, resulting in a 26% margin[59] Market and Strategy - The total addressable market (TAM) for observability is $14 billion, and for security is $85 billion[27] - Dynatrace has a go-to-market strategy targeting 15,000 customers across strategic enterprise and commercial segments[29]