Oleksandr Kuzmenko, EditorPublished 6 min readAI-assisted · editor-reviewed
Why it matters
Audit your multi-document agent architectures for hidden assumptions before deploying autonomous agents on ambiguous enterprise data.
TL;DR
- 01Top frontier models reached a maximum pass rate of 10.8% on complex multi-document tax environments.
- 02Agents frequently execute calculations accurately but fail to detect unstated constraints and pending legal deadlines.
- 03Heterogeneous workspaces (Excel, scanned receipts, email chains) trigger reasoning breakdowns that single-prompt tests miss.
Key facts
- Claude Sonnet 5.5 Pass Rate
- 10.8% (self-reported by Surge AI)
- Claude Fable 5.1 Pass Rate
- 9.6% (self-reported by Surge AI)
- Claude Opus 5.5 Pass Rate
- 9.2% (self-reported by Surge AI)
- Workspace Files in Benchmark
- 1,789 files across 50 environments
- Rubric Pass Threshold
- 95% completion criteria
Beyond Synthetic Question-Answering
Most statutory and financial LLM benchmarks present isolated questions with complete context provided in the prompt. Surge AI's DAYJOB: Tax shifts testing toward real practitioner workloads where the initial prompt is incomplete or misdirected. Agents are tasked with generating filings, workpapers, client memos, and audit reviews across 50 production environments.
Workspace Scale and Heterogeneous Evidence
The benchmark environments aggregate 1,789 workspace files, with assignments containing a median of 30 files and up to 123 distinct documents. These test suites include:
- Document Types: 1,487 PDFs, 117 Excel spreadsheets, 38 Word documents, along with JSON feeds, CSVs, correspondence emails, and images.
- Unstructured Artifacts: Scanned photographs of handwritten merchant receipts requiring inventory reconciliation against general ledgers.
- Multi-Jurisdiction Clues: Field dispatch logs showing crew assignments across five states, disproving client claims of single-state operations.
Leaderboard Results and Failure Modes
Evaluations measured pass@1 against a strict 95% rubric threshold requiring accurate calculations, evidence citations, and identification of unstated consequential issues:
- Claude Sonnet 5.5: 10.8% pass rate (highest measured).
- Claude Fable 5.1: 9.6% pass rate.
- Claude Opus 5.5: 9.2% pass rate.
Frontier agents regularly failed to flag critical expiration deadlines or investigate unverified assumptions unless explicitly instructed to search for them, underscoring that raw model reasoning cannot yet replace human-in-the-loop review for compliance-heavy domains.
✓ When to use
- Benchmarking autonomous agents deployed on unorganized document archives, corporate ledgers, or multi-jurisdiction data.
- Stress-testing model capabilities in identifying implicit constraints without explicit user prompts.
✕ When NOT to use
- Evaluating basic zero-shot code completion or straightforward syntax generation.
- Pipelines where user input is strictly schema-validated and contains zero unstated edge cases.
What to do today
- Incorporate uncurated multi-file directories into your agent test suites rather than relying on clean markdown prompts.
- Add explicit verification steps in agent prompts to question underlying assumptions before executing tool workflows.
What the community says
“In the U.S, in the eyes of the IRS, you are liable for your return. But you can sue the accountant in civil court, and you likely won't have to if you can prove to them an error was made.”
“I prepare my own tax returns... and have for the past few years, fed my draft return along with all the supporting paperwork into Claude. It has: 1. found errors... 2. found items which would appear to be errors except for context”