Quantifying Agentic Code Sloppiness with AST-Grep and Erosion Metrics
Benchmarking reveals AI coding agents generate code with roughly double the verbosity and architectural erosion of human repositories. Iterative agent loops accumulate poor structural abstractions over time, causing strict multi-checkpoint benchmark success rates to collapse to zero percent.
Impact: High
Why it matters
Greenfield unit tests can verify code correctness but completely fail to measure codebase decay. As developers rapidly generate code with LLMs, structural degradation builds up silently until even frontier models fail multi-step tasks.
TL;DR
- 01LLMs produce code with approximately double the verbosity and architectural erosion of established human codebases.
- 02Frontier models reach a 0% pass rate under strict multi-round iterative evaluations where context is cleared between checkpoints.
- 03LLM-as-a-judge quality evaluations are fundamentally unstable and often equivalent to random guessing when ranking code quality.
Key facts
- Agent Code Verbosity
- 0.33 ± 0.10 (vs 0.15 ± 0.06 in human repos)
- Agent Code Erosion
- 0.68 ± 0.20 (vs 0.31 ± 0.17 in human repos)
- Observed Vibe Projects Verbosity
- Up to 0.40
- Observed Vibe Projects Erosion
- Up to 0.75
- Strict Multi-Round Solve Rate
- 0% across frontier reasoning models
Measuring the Structural Cost of Vibe Coding
While large language models write syntactically valid code that passes greenfield tests, iterative coding prompts create an explosion of lines of code. Sebastian at Earendil analyzed how to move beyond vibes and LLM-as-a-judge evaluations—which often behave like random number generators—by testing formal quantitative metrics from the SlopCodeBench benchmark.
Verbosity and Function Erosion
SlopCodeBench isolates AI-generated bloat using two mathematical metrics:
- Verbosity: The ratio of duplicate lines and
ast-grepflagged pattern violations over total lines of code (LOC). Human code averages 0.15 ± 0.06, while agent-written code reaches 0.33 ± 0.10. - Erosion: The concentration of codebase mass inside overly complex functions where cyclomatic complexity
CC(f) > 10. Function mass is calculated asmass(f) = CC(f) * sqrt(SLOC(f)). Agent code scores 0.68 ± 0.20 on erosion, more than double the human baseline of 0.31 ± 0.17.
Why Iterative Agents Fail Long Horizon Tasks
In SlopCodeBench evaluations, agents face multi-round instruction checkpoints where context is cleared between iterations, reflecting realistic developer sessions. Bad structural abstractions compound across cycles. Under strict pass criteria across all checkpoints, state-of-the-art models drop to a 0% solve rate, demonstrating that agents struggle to deal with accumulated structural slop on their own.
Try it in 2 minutes
mass = cc * (sloc ** 0.5)
erosion = sum(f.mass for f in funcs if f.cc > 10) / sum(f.mass for f in funcs)python
✓ When to use
- Use AST-based pattern matching and erosion formulas when assessing long-term health in heavily AI-generated repositories.
✕ When NOT to use
- Do not rely solely on naive LLM-as-a-judge numerical prompts (1-10) to evaluate code cleanliness.
What to do today
- Monitor delta in lines of code (LOC) across agentic PRs as an early warning sign of code bloat.
- Track cyclomatic complexity and function mass to detect erosion in critical codebase paths.
Sources