Metrics
Search documents
Leadership in AI Assisted Engineering – Justin Reock, DX (acq. Atlassian)
AI Engineer· 2025-12-19 18:14
Impact of Generative AI - Industry averages show modest positive indicators with a 75% increase in documentation quality and a 34% increase in code quality [6] - Some companies experience up to 20% increases in change confidence, while others see 20% decreases, highlighting extreme volatility [8] - An increase of 2% in change failure rate, against an industry benchmark of 4%, could mean shipping up to 50% more defects [9] Strategies for Successful AI Adoption - Top-down mandates for AI adoption are ineffective; lack of education and enablement negatively impacts adoption [9][10][11] - Clear AI policies and dedicated time for learning and experimentation are crucial for moving the needle [12][13] - Integrating AI across the Software Development Life Cycle (SDLC) and addressing bottlenecks beyond just code completion is essential [13][14] - Open discussions about metrics and evangelizing wins are necessary to reduce the fear of AI and ensure employee success [15][16] Metrics and Measurement - Focus on speed and quality metrics, including PR throughput, change failure rate, change confidence, and maintainability [21][22] - Utilize telemetry metrics, experience sampling, and effective surveys to gather comprehensive data [22][23][24][25] - Implement a DXAI measurement framework, considering utilization, impact, and cost to assess AI maturity [28][29] Compliance and Trust - Establish feedback loops for system prompts to ensure the output is trustworthy and aligned with organizational standards [33][34][35] - Understand and control the temperature setting to manage the determinism and non-determinism of AI models [35][36][37] Employee Success - Provide education and adequate time for developers to learn and experiment with AI, focusing on valuable use cases like stack trace analysis [40][41][42][43] - Leverage self-hosted and private models, partner with compliance from the start, and think creatively to unblock AI usage [44] Optimizing the SDLC - Identify and fix bottlenecks in the SDLC, as time saved on non-bottleneck areas is worthless [45][46] - Learn from examples like Morgan Stanley, which saves 300,000 hours annually by using AI to modernize legacy code [47][48] - Emulate Zapier's approach by using AI to enhance onboarding and increase the effectiveness of new engineers [49][50]
100M views… Zero Impact?
20VC with Harry Stebbings· 2025-10-31 16:27
I've had clips that have exceeded 100 million views and the branding of the show is there. It's clearly an interview on my podcast. There's a link to the full episode and the impact on the long- form episode has been imperceptible.You look at the downloads and there is sweet all zero impact. So, I think it's also just worth noting these teams and the platforms, you know, the teams behind the platforms are very smart. They're very well funded.They have data scientists and teams of many others whose sole obje ...
X @Starknet (BTCFi arc)
Starknet 🐺🐱· 2025-10-02 13:44
RT Brother Lyskey (@0xLyskey)1/ Starknet metrics since BTCFi launch.no blabla, just metrics 🧵 https://t.co/d1FHJZ6Uuk ...
Meta's miss: Audio Rooms
20VC with Harry Stebbings· 2025-09-07 14:01
Goal Setting - The North Star is the goal, not a metric [1] - The company's goal should be clearly defined [1] - A metric is used to describe the goal, but it is never a perfect representation [1] Metric Definition - The most important thing is to have absolute clarity on what your goal is and then do the best you possibly can to describe that goal with a metric [1] - A metric is always broken [1] Example - When joining Meta, the goal was to connect the world online [1]
The truth about North Star metrics
20VC with Harry Stebbings· 2025-09-06 14:00
Core Goal & Metrics - The North Star is the company's goal, not a metric [1] - The company's goal should be clearly defined [1] - Metrics are used to describe the goal, but never perfectly [1] - Metrics are inherently flawed [2]
