NVIDIA SWE-Serve: About One in Three Agent Patches Fail Live Serving
NVIDIA, with input from the SGLang team, released SWE-Serve, a benchmark of 53 inference-engineering tasks derived from 83 merged SGLang pull requests. Across 19 tasks with live-serving checks, the same patches pass 69.4% of the time when those checks are excluded but only 45.9% with the complete verifier — about one in three patches that pass the other tests fail live-serving validation.
Why it matters
NVIDIA, with input from the SGLang team, released SWE-Serve, a benchmark of 53 inference-engineering tasks derived from 83 merged SGLang pull requests. Across 19 tasks with live-serving checks, the same patches pass 69.4% of the time when those checks are excluded but only 45.9% with the complete verifier — about one in three patches that pass the other tests fail live-serving validation.