Debugging Deterministic Citation Verification in AI Research Agents
A deterministic citation checker failed on nearly half of valid AI agent quotes due to subtle text-extraction bugs. Cleaning HTML tags and footnote spacing restored quote validation accuracy while catching genuinely hallucinated URLs.
Impact: Medium
Why it matters
You can prevent validation code from falsely rejecting accurate model outputs by testing verification pipelines against known-good inputs before deploying retry loops.
TL;DR
- 01Direct string matching in verification gates fails when HTML parsing corrupts whitespace and bracket formatting.
- 02Testing verifiers against mutated inputs guarantees they retain high selectivity without false rejections.
- 03LLMs routinely generate plausible-sounding URL paths that do not exist on allowed domains.
Key facts
- Initial False Rejections
- 12 out of 18 quotes
- Post-Fix Match Rate
- 18 out of 18 valid quotes
- Fabricated URLs Identified
- 6 out of 10 dead links
- Batch Execution Cost
- $0.037 for 37 API calls
The Pitfalls of Naive Text Normalization
Deterministic verification gates enforce that every quote proposed by an LLM exists byte-for-byte in the fetched document. However, standard regex rules like <[^>]+> break when encountering embedded JSON inside HTML attributes that contain > characters. Furthermore, converting tags directly into whitespace turns inline brackets like [1] into [ 1 ], causing exact string matching to fail on perfectly quoted content.
Fixing Extraction Logic and Mutation Testing
To eliminate false rejections, the text extractor must strip attribute values before removing HTML tags, handle unspaced inline tags, and trim embedded raw TeX formatting. Verifying the check requires mutation testing: running altered quotes through the pipeline to ensure it catches real hallucinations without lowering the acceptance threshold.
Detecting Upstream Source Hallucinations
Once downstream text matching is calibrated, the gate effectively flags fake links. In testing across 20 questions, 6 out of 10 rejected URLs were fabricated paths (e.g., non-existent documentation pages). Restricting the agent to an explicit domain allowlist prevents arbitrary network fetches while keeping quote validation deterministic.
Try it in 2 minutes
const ALLOW = ['en.wikipedia.org', 'developer.mozilla.org', 'docs.python.org', 'datatracker.ietf.org'];
function sanitizeText(html) {
return html.replace(/\s+=\s*"[^"]*"/g, '').replace(/<[^>]+>/g, ' ').replace(/\[\s+(\d+)\s+\]/g, '[$1]');
}javascript
✓ When to use
- Building RAG applications that require verified direct citations from primary documentation.
- Implementing deterministic guardrails upstream of automated report generation.
✕ When NOT to use
- Generating creative writing or loose summaries where exact string matching is unnecessary.
What to do today
- Test deterministic verification code against known-good string outputs before enabling retry prompts.
- Audit text-cleaning regexes to ensure attribute values and brackets do not corrupt raw text matches.
Sources