CrucibleBench Evaluates AI Agents with Persistent Text MUDs and Uncovers LLM Judge Flaws
CrucibleBench uses persistent text dungeons to benchmark LLM agent behaviors in complex social environments. The evaluation reveals that LLM-as-a-judge scoring reorders leaderboards by up to six positions while masking instability, and uncovers looping bugs across frontier models.
Impact: High
Why it matters
Engineers can audit agent behavioral failure modes and replace unreliable LLM judges with deterministic state-machine telemetry.
TL;DR
- 01LLM judges can silently shift leaderboard rankings by six positions due to low classifier agreement.
- 02Dialogue looping affects up to 66% of frontier AI agent runs when persistent state demands conversational adaptation.
- 03State-machine telemetry provides explicit detection of hallucinated API calls without probabilistic LLM judges.
Key facts
- Rooms
- 12
- Items
- 14
- NPCs
- 4
- Dialogue looping frequency
- 14-66% of frontier runs
- Judge agreement range
- 21.7% to 84.8%
- Benchmark dataset size
- 650 replayable transcripts
Persistent State Environments for Agent Benchmarking
CrucibleBench tests models inside a text dungeon featuring 12 rooms, 14 items, and 4 NPCs with trust and suspicion states scaled from 0 to 100. Because items taken stay taken and actions are bounded by explicit command parsers, hallucinated API calls or invalid world transitions are detected algorithmically.
Uncovering LLM-as-a-Judge Instabilities
The study reveals that integrating an LLM classifier into the scoring harness alters model leaderboards by up to six positions. Agreement between the scoring judge and an independent judge ranged from 21.7% to 84.8%, despite aggregate inter-rater reliability showing a misleading kappa score of 0.04. Removing judge-dependent dimensions shifted model positions significantly.
Algorithmic Failure Modes in Frontier Models
Telemetry tracking revealed three major behavioral bugs across 50 runs per model:
- Dialogue Looping: Occurred in 14% to 66% of frontier model runs, where agents repeatedly issued
talkcommands without altering conversational strategy. - Wrong-Room Interactions: Grok 4 demonstrated a 12% incidence of addressing non-existent entities in empty rooms.
- Exploration Paralysis: Models repeatedly executed
lookcommands or failed to move beyond two rooms in over 20 turns.
Try it in 2 minutes
git clone https://github.com/cruciblebench/cruciblebench.git && cd cruciblebench && python eval.py --model anthropic/claude-3-5-sonnetbash
✓ When to use
- When designing behavioral benchmarks for multi-turn AI agents in stateful environments.
- When auditing LLM-as-a-judge reliability and ranking sensitivity in evaluation harnesses.
✕ When NOT to use
- Not for measuring simple single-turn factual recall or code syntax generation.
- Not as a single definitive measure of general intelligence without domain calibration.
What to do today
- Audit internal LLM-as-a-judge evaluation pipelines by measuring per-subject judge agreement.
- Add state-machine loop detection to production AI agent harnesses to break repetitive tool loops.
- Download CrucibleBench's 650 transcripts to analyze frontier agent conversational failure modes.
What the community says
“Learn about agent loops, prompting iteration, etc. I vibe coded a few room proof of concept MUD. I added a tools called oracle that it could ask me questions to help along its way.”
Sources