Oleksandr Kuzmenko, EditorPublished 6 min readAI-assisted · editor-reviewed
TL;DR
- Token spend is an ineffective proxy for developer productivity, varying up to 240x for identical commit volumes.
- Doubling an engineer's token spend correlates with only ~10% more commits.
- In nearly 9 out of 10 technical agent evaluations, agents are used for tasks beyond coding, such as analysis, research, and planning.
- Organizations should limit token metrics to resource accounting and evaluate AI work based on tasks and outcomes.
Key facts
- Engineers Tracked
- 404 JetBrains engineers
- Survey Participants
- 412 employees
- Observation Window
- 5 weeks
- Spend Spread at Equal Commits
- 39x to 240x (90th vs 10th percentile)
- Commit Gain per 2x Spend
- ~10% more commits
- Non-Coding Agent Usage
- Nearly 9 out of 10 evaluations
Source: Key facts — Survey Participants, Observation Window
Source: Key facts. Survey Participants: 412 employees. Observation Window: 5 weeks. Each figure is copied from the story.
- Survey Participants412 employees
- Observation Window5 weeks
The Disconnect Between Token Spend and Commits
Tracking five weeks of token spend across 404 JetBrains engineers revealed that token volume acts as an unreliable proxy for software output. For engineers producing identical weekly commit counts, the 90th percentile of token consumption was 39 to 240 times higher than the 10th percentile. Furthermore, within the same engineer, doubling token spend was associated with only an approximate 10% increase in commit volume.
What Explains the Token Variance?
Standard engineering metrics accounted for only 38% of the variation in spend between different engineers, and a mere 13% of week-to-week variation for an individual developer. High spenders remained high spenders across all five weeks regardless of their commit velocity. Token spend measures how much engineers use agents across many kinds of work, not the outcome and value of that work.
Beyond Code: Agents for Analysis, Research, and Planning
A survey of 412 JetBrains employees demonstrated why code-centric metrics fail: in nearly nine out of ten agent evaluations, respondents in technical roles reported using agents for tasks beyond direct coding, such as analysis, research, and planning. The authors advise engineering organizations to limit token metrics to resource accounting and evaluate agent-assisted work using evidence about tasks and outcomes.
When to use
- Use token metrics when managing cloud infrastructure costs, establishing organization budgets, or monitoring overall AI tool adoption.
When NOT to use
- Do not use token spend to rank developer performance or evaluate code velocity.
What to do today
- Stop using token volume or cost per engineer as a software productivity metric.
- Restructure engineering AI accounting to treat tokens purely as a cloud compute budget line.
- Assess coding agent value based on specific task completions, research efficiency, and engineering outcomes.
Step names are taken from What to do today.
Step names are taken from What to do today. Stop using token volume or cost per engineer as a software productivity metric.. Restructure engineering AI accounting to treat tokens purely as a cloud compute budget line.. Assess coding agent value based on specific task completions, research efficiency, and engineering outcomes.
- Stop using token volume or cost per engineer as a software productivity metric.
- Restructure engineering AI accounting to treat tokens purely as a cloud compute budget line.
- Assess coding agent value based on specific task completions, research efficiency, and engineering outcomes.