2 min · Hugging Face Blog
XYBench Reveals Frontier Language Models Fail to Correct Flawed User Assumptions
Carnegie Mellon, NVIDIA, and Johns Hopkins launched XYBench, an 8,115-question benchmark evaluating whether models recognize misguided technical queries. Results show top frontier models focus up to 57% of answers on flawed literal requests rather than underlying architectural solutions.
Why it matters. LLM agents serving as developer assistants often implement fragile workarounds requested by users rather than addressing the actual technical goal.
