Automating Ground-Truth Extraction with Dual-LLM Gating and Agent Arbitration
The dnb-toc-ground-truth project introduces a hybrid pipeline that pairs two independent vision LLMs with a 90% agreement threshold for structured data extraction. Mismatches automatically escalate to a multimodal agent like Claude for arbitration, creating high-quality datasets to fine-tune lightweight models.
Impact: Medium
Why it matters
You can drastically cut structured extraction costs by reserving expensive frontier LLM agents strictly for resolving disagreement between smaller vision models.
TL;DR
- 01Use dual-LLM consensus to automatically filter high-confidence extraction results.
- 02Escalate low-confidence or conflicting OCR results to multimodal agents.
- 03Fine-tune smaller models like NuExtract3 on agent-verified ground truth.
Key facts
- Auto-gating match threshold
- 90%
- Evaluated books sample size
- 54 of 547
- Crossref evaluation mean F1 score
- 76%
- Crossref evaluation mean precision
- 85%
- Crossref evaluation mean recall
- 77%
Dual-Model Agreement Gating
To generate machine-checkable ground truth at scale, the pipeline runs two distinct vision LLMs—such as Qwen/Qwen3-Omni-30B-A3B-Instruct and mistralai/Mistral-Small-3.2-24B-Instruct-2506—on identical table-of-contents page images. The outputs are diffed using dnb_toc_ground_truth.matching.gate_books. If the top agreeing pair meets or exceeds a 90% entry match rate, the merged JSON is written directly (source: bulk_gate).
Agent Arbitration and LoRA Distillation
When the initial agreement gate fails or models disagree, the pipeline invokes a multimodal AI agent like Claude (source: agent_arbitration). The agent reviews the raw page images and transcribes the ground truth. This synthetic ground truth is then harvested to LoRA fine-tune smaller task-specific extraction models like numind/NuExtract3.
External Verification via Crossref
To validate extraction quality independently of LLM self-consistency, cli/backfill_crossref.py and cli/evaluate_crossref.py cross-check ground truth against Crossref per-chapter metadata. Across an initial evaluation sample of 54 books with complete metadata, the ground truth achieved a mean precision of 85%, recall of 77%, and F1 score of 76%.
Try it in 2 minutes
uv run python cli/evaluate_crossref.py --all-modelsbash
✓ When to use
- When building high-volume structured data extraction pipelines from scanned PDFs or images.
- When creating fine-tuning datasets for small specialized models at minimal API cost.
✕ When NOT to use
- When extracting simple text documents that do not require multi-modal visual alignment.
- When API budget allows running all documents directly through top-tier frontier agent models.
What to do today
- Run uv run python cli/evaluate_crossref.py to test extraction evaluation pipelines.
- Implement a 90% diff threshold between dual vision LLMs before invoking costly agent calls.
Sources