Tracelint Replaces LLM Judges with Deterministic OpenInference Checks in CI
Tracelint is a lightweight open-source tool that validates agent OpenInference execution traces without calling LLM judges. It detects broken schemas, non-existent tools, and side-effect actions taken after failed pipeline steps, returning standard non-zero exit codes for CI/CD pipelines.
Why it matters
You can catch agent regressions and invalid tool execution loops in GitHub Actions without paying LLM inference costs or tolerating judge flakiness.
TL;DR
- 01Use deterministic trace rules for structural tool checks instead of expensive LLM judges.
- 02Declare side-effecting tools and failure pointers to fail CI with exit code 2 when broken data is passed.
- 03Candidate anomalies like loops are isolated to avoid noisy false positives in automated builds.
Key facts
- Supported Telemetry Standard
- OpenInference (Phoenix, LangGraph compatible)
- Exit Code on Hard Defect
- Exit code 2
- Validation Mechanism
- Deterministic static checking without LLM inference calls
Eliminating Flaky LLM Judges in CI
Agent evaluation in production pipelines often suffers from non-deterministic scoring and high token overhead when delegating validation to external language models. Tracelint runs static validation against execution spans adhering to the OpenInference standard. If a trace records that an automated pipeline finished with UNSTABLE or FAILURE status, and an agent subsequently passes that build identifier to a destructive action such as deploy, tracelint catches the contract breach immediately.
Deterministic Rules versus Review Candidates
The tool classifies findings into three distinct operational states:
- Hard defects: Definitively broken traces, such as arguments violating tool JSON Schemas, calls to nonexistent tools, or reusing failed step output into actions marked with
"side_effecting": true. The CLI exits with code2to halt automated pull request merges. - Candidates: Anomalies that might represent legitimate retries or polling, such as repeated identical invocations without progress. These are logged for manual review without failing the build.
- Not checked: Steps lacking an explicit
failure_whenpredicate, preventing false passes.
Initial schema profiles can be generated directly from recorded traces using tracelint init spans.json --format openinference -o tools.json.
Try it in 2 minutes
tracelint check spans.json --format openinference --tools tools.jsonbash
✓ When to use
- Enforcing safety contracts before agents invoke side-effecting operations like database writes or deployments.
- Running fast, repeatable unit tests for AI agent actions inside CI pipelines.
✕ When NOT to use
- Evaluating semantic response quality or contextual politeness that requires nuanced natural language understanding.
- Workflows that lack structured OpenInference telemetry or tool JSON schemas.
What to do today
- Export agent trace telemetry using the OpenInference standard in existing test fixtures.
- Generate initial rule definitions using tracelint init.
- Add tracelint check spans.json to your continuous integration pipeline.