Evaluating LLM Token Efficiency and Correctness Across Programming Languages
Dan Luu benchmarks how programming languages affect LLM token usage and solution correctness on non-trivial tasks. Contrary to claims that concise dynamic languages save up to 2.6x on tokens, real-world task evals show mainstream dynamic and static languages perform similarly, with mainstream languages outperforming obscure concise ones.
Impact: Medium
Why it matters
Stick with mainstream languages for AI coding workflows rather than switching to dense esoteric languages for hypothetical token savings.
TL;DR
- 01Trivial benchmarks drastically overstate token efficiency gains for dynamic and concise languages.
- 02Complex coding tasks neutralize token cost differences between dynamic and static languages.
- 03Model performance strongly correlates with programming language popularity in public datasets.
Key facts
- Trivial benchmark efficiency gap
- 2.6x (C vs Clojure)
- Eval task
- zstd decoder RFC implementation
- Evaluated model
- GPT-5.6 Sol
Trivial Benchmarks Distort Token Consumption Reality
Popular comparisons claim concise dynamic languages like Clojure require only 109 tokens versus 2.6x more in C for equivalent tasks. However, these benchmarks rely on trivial Rosetta Code problems. On tiny scripts, token count is dominated by boilerplate and printing logic, failing to reflect production software development.
Task Complexity Dilutes Language Advantage
When evaluating agents tasked with implementing a complete zstd decoder RFC using GPT-5.6 Sol across medium and ultra reasoning efforts, language type boundaries collapse. While dynamic languages cluster slightly better at medium effort, static languages deliver equal or superior correctness and cost efficiency at ultra effort.
Popularity Correlates with Correctness
Obscure or dense languages like J fail to sustain token efficiency on larger tasks. Evaluation data shows a positive correlation between language popularity and correctness, as frontier models benefit from vast training corpora and targeted synthetic RL environments.
✓ When to use
- When choosing a programming language stack for AI-assisted or agent-driven software development.
✕ When NOT to use
- When evaluating macro token costs across an entire engineering organization rather than individual prompt tasks.
What to do today
- Keep primary AI coding tasks in mainstream, highly represented languages like Python, TypeScript, Rust, or Go.
- Avoid switching codebases to dense obscure languages solely for token reduction promises.
Sources