Oleksandr Kuzmenko, EditorPublished 6 min readAI-assisted · editor-reviewed
TL;DR
- XYBench benchmarks LLM responses across 8,115 questions exhibiting the XY problem.
- Frontier models prioritize underlying goals over mistaken requests in only 35-41% of Stack Overflow responses.
- Supplying latent goals improves solution inclusion but can degrade explicit misconception warnings.
Key facts
- Benchmark Scale
- 8,115 real-world questions
- Primary Data Sources
- 1,557 Stack Overflow, 6,272 WikiHow
- Academic Award
- Best Resource Paper Award at EMNLP 2026
- Stack Overflow Goal Focus
- 35-41% focus on actual goal vs 47-57% on literal request
The Pragmatic Gap in Agent Troubleshooting
AI coding assistants regularly waste hundreds of generation tokens implementing dead-end workarounds because they take user prompts literally. To quantify this failure, researchers from Carnegie Mellon University, NVIDIA, and Johns Hopkins developed XYBench, which was awarded Best Resource Paper at EMNLP 2026.
The benchmark contains 8,115 questions curated from Stack Overflow (1,557), Stack Exchange (184), manual expert contributions (102), and procedural WikiHow guides (6,272). Each sample maps a stated request Y against the latent intended goal X.
Model Performance Breakdown
Evaluated models—including Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro—displayed a systematic bias toward literal compliance:
- Stated Request (Y) Solutions: Present in 75% to 86% of Stack Overflow responses.
- Underlying Goal (X) Solutions: Addressed in only 59% to 71% of responses.
- Primary Focus: Models prioritized the underlying goal in only 35% to 41% of completions, while dedicating the majority of context space to the mistaken literal request in 47% to 57% of cases.
Why Intent Injection Isn't Enough
Supplying both the stated request and the intended request directly to Claude Sonnet 4.6 raised its solution inclusion rate from 59% to 67%, but reduced explicit misconception identification from 47% to 34%. Production agent pipelines must explicitly instruct models via system instructions to interrogate developer assumptions before initiating code edits.
