Agents & MCP
NVIDIA Releases SkillEvaluator to Benchmark AI Agent Skills in Claude Code and Cursor
NVIDIA open-sourced SkillEvaluator, an automated testing tool for AI agent skills across Claude Code, Codex, and Cursor. Evaluation across 300+ verified skills demonstrated an average 31-point improvement in task correctness and efficiency.
August 20, 2026 4 min read
Curated by Oleksandr Kuzmenko, AI Product EngineerUpdated August 20, 2026Sources cited on every story
AI-assisted · editor-reviewedHow we use AI
Impact: High
Why it matters
You can benchmark your custom agent skills and Model Context Protocol setups using a repeatable 3-tier evaluation framework.
TL;DR
- 01Use SkillEvaluator to measure performance lift for custom agent skills before deployment.
- 02Isolate evaluation runs using Harbor sandboxes to prevent side effects during testing.
- 03Ensure distinct tool descriptions in Tier 2 to avoid confusion between overlapping agent skills.
Key facts
- Tested Skills Count
- 300+ across 30+ products
- Average Overall Skill Lift
- 31 points (39 points ex. security)
- Supported Agent Harnesses
- Claude Code, Codex, Cursor
Three-Tier Agent Skill Benchmark
NVIDIA SkillEvaluator standardizes agent skill testing through three evaluation tiers:
- Tier 1 (Safety & Structure): Runs static checks, frontmatter validation, prompt injection detection, secret scanning, and script linting.
- Tier 2 (Distinctiveness): Uses embedding similarity to identify overlapping guidance or duplicated tool descriptors across catalogs.
- Tier 3 (Live Sandbox Evaluation): Executes task runs inside isolated Harbor sandboxes, measuring with-skill versus without-skill performance.
Benchmark Results across 300+ Skills
Evaluations conducted across Claude Code and Codex harnesses demonstrate substantial gains over unassisted baselines:
- Correctness: Baseline without skill is
46/100. - Discoverability: Baseline is
42/100. - Effectiveness: Baseline is
39/100. - Efficiency: Baseline is
43/100. - Overall Skill Lift: Average gain of
31 pointsacross evaluated dimensions (39 pointsexcluding security).
Try it in 2 minutes
skillevaluator create-eval-dataset ./my-skill --full
skillevaluator tier3 evaluate ./my-skillbash
✓ When to use
- When building custom tools or skills for Claude Code, Codex, or Cursor.
- When validating prompt injection and security compliance for agent integrations.
✕ When NOT to use
- When making simple one-off API calls without agent harnesses or tool routing.
What to do today
- Generate an eval dataset for your custom agent skill using skillevaluator create-eval-dataset.
- Run Tier 3 live evaluations to benchmark execution accuracy and token consumption.
Sources