Oleksandr Kuzmenko, EditorPublished 6 min readAI-assisted · editor-reviewed
TL;DR
- Pass^k measures all-run reliability, uncovering substantial failure rates hidden by Mean@k averages.
- Flat token distributions cause execution flips at temperature 0.0 due to hosted runtime noise.
- ALTK-Evolve extracts targeted consistency guidelines via black-box resampling to halve the consistency gap.
Key facts
- Base Mean@5 (GPT-4.1 AppWorld)
- 77.4%
- Base Pass^5 (GPT-4.1 AppWorld)
- 53.0%
- Consistency Gap
- 24.4 percentage points
- Post-Guideline Pass^5
- 69.0%
- Post-Guideline Consistency Gap
- 12.0 percentage points
Source: Key facts — Base Mean@5 (GPT-4.1 AppWorld), Base Pass^5 (GPT-4.1 AppWorld), Post-Guideline Pass^5
Source: Key facts. Base Mean@5 (GPT-4.1 AppWorld): 77.4%. Base Pass^5 (GPT-4.1 AppWorld): 53.0%. Post-Guideline Pass^5: 69.0%. The chart compares only values that share a unit. Each figure is copied from the story.
Metric chart
- 77.4%Base Mean@5 (GPT-4.1 AppWorld)
- 53.0%Base Pass^5 (GPT-4.1 AppWorld)
- 69.0%Post-Guideline Pass^5
- Base Mean@5 (GPT-4.1 AppWorld)77.4%
- Base Pass^5 (GPT-4.1 AppWorld)53.0%
- Post-Guideline Pass^569.0%
Source: Key facts — Consistency Gap, Post-Guideline Consistency Gap
Source: Key facts. Consistency Gap: 24.4 percentage points. Post-Guideline Consistency Gap: 12.0 percentage points. The chart compares only values that share a unit. Each figure is copied from the story.
- 24.4 percentage pointsConsistency Gap
- 12.0 percentage pointsPost-Guideline Consistency Gap
The Hidden Consistency Gap in AI Agents
Standard evaluations track Mean@k, but production systems demand Pass^k—the guarantee that an agent solves a task on every attempt. Evaluating a ReAct agent driven by GPT-4.1 on the AppWorld test_normal benchmark revealed an aggregate Mean@5 of 77.4%, but a Pass^5 of only 53.0%. This leaves a 24.4-point consistency gap where agents succeed or fail unpredictably on identical inputs.
Why Greedy Decoding Fails to Enforce Determinism
Even at temperature 0.0, agents drift due to hardware perturbations, GPU floating-point non-associativity, and hosted batching. When next-token probability distributions are flat across multiple valid tool arguments, minor numerical noise flips the execution trajectory.
Single-Trace Offline Resampling
ALTK-Evolve's Consistency Analyzer identifies fragile steps from a single execution trace without labeled ground truth: 1. Replays each decision step offline with a single model call requesting k=5 completions. 2. Computes a black-box consistency score based on output variations across resamples to pinpoint steps at risk of flipping. 3. Synthesizes targeted guidelines (e.g., anchoring regex markers or verifying multi-record queries).
Injecting these dynamic guidelines into the prompt narrowed the AppWorld consistency gap from 24.4pp to 12.0pp while raising Pass^5 to 69.0%.
When to use
- Mission-critical agentic workflows like financial reconciliation or legal compliance checking where every run must succeed.
When NOT to use
- Simple single-turn tasks where agent branching and multi-step tool calls are absent.
What to do today
- Audit existing ReAct agents using Pass^k across at least 5 identical runs.
- Implement black-box k-completion resampling on recorded traces to detect unstable decision points.
Step names are taken from What to do today.
Step names are taken from What to do today. Audit existing ReAct agents using Pass^k across at least 5 identical runs.. Implement black-box k-completion resampling on recorded traces to detect unstable decision points.
- Audit existing ReAct agents using Pass^k across at least 5 identical runs.
- Implement black-box k-completion resampling on recorded traces to detect unstable decision points.