Why 'Percentage of Code Written by AI' is a Vanity Metric
Industry claims about volume of AI-written code often mask the lack of actual outcome-based productivity gains. Engineering leads should favor DORA metrics and reliability over volume-based vanity scores.
Impact: Medium
Why it matters
Vanity metrics like 'lines of code' or 'AI-generated percentage' can mislead leadership and result in poor headcount planning.
TL;DR
- 01Avoid measuring engineering success by volume of code generated.
- 02Re-assert focus on DORA metrics and customer value outcomes.
- 03Question whether vendor claims track output or merely input volume.
The Metrics Trap
Marketing claims have shifted from 'completed tasks faster' (GitHub Copilot's 55% claim) to 'percent of code written by AI'. This shift focuses on volume, not value. Metrics like these move budgets and performance expectations without proven outcome causality.
What to Track Instead
- DORA Metrics: Focus on deployment frequency and change failure rates.
- Reliability: Measure system uptime and incident response.
- Outcome Evidence: Prioritize revenue, customer conversion, and MAU over code generation volume.
The Reality of Layoffs
Companies citing AI as a reason for workforce reduction rarely demonstrate that underutilized capacity was the root cause. Without evidence, these decisions appear to be based on vanity metrics rather than operational efficiency.
✕ When NOT to use
- Using LOC as a KPI
- Justifying layoffs purely via AI code output