Results for “Benchmarks”

Why it matters

Daily token usage on OpenRouter reveals that Nous Research's Hermes Agent has surpassed OpenClaw, processing 458 billion daily tokens versus OpenClaw's 173 billion. The usage flip demonstrates how persistent memory architectures drastically reduce repeat context costs compared to session-native designs.

Why it matters

Zhipu AI's GLM 5.2 model demonstrates competitive coding performance and strong vulnerability detection capabilities as an open-weight MoE model.

Why it matters

DeepMind released EmbeddingGemma 2, a 740M-parameter multimodal embedding model under the Apache 2.0 license. It unifies code, text, image, audio, and video representations into a single vector space with an 8K token context window.

Why it matters

Codex's background computer-use automation engine runs as an unbundled local Model Context Protocol server on macOS. Claude Code can directly call this server headless to click and drag without stealing the user's cursor.

Why it matters

Artificial Analysis published its Coding Agent Index evaluating model and harness combinations. Claude Sonnet 5.5 in Claude Code leads with 68 at $14.19 per task, while GPT-6.1 Sol in Codex achieves 63 for $1.04.

Why it matters

Driftproof benchmarked Claude Opus 5.5 within 15 hours of release across three core agent skills from addyosmani/agent-skills. Without explicit SKILL.md guidance, Opus 5.5 regressed on conventional commit prefixes and review severity labels compared to Opus 5. Calling the new model in Claude Code also strictly requires upgrading to version 2.1.280 or newer.

Why it matters

Prism ML released Ternary Bonsai 2 27B, quantizing Qwen3.8 27B down to 1.76 bits per weight at a 5.9GB memory footprint. The release retains 98.2% of full-precision benchmark performance while running local agentic coding loops on consumer GPUs.

Why it matters

The Allen Institute released TutorMoments, an open framework designed to measure whether LLMs know when to assist versus when to let the user work independently. Evaluating seven leading models showed that evaluation-aware prompts significantly improve agent decision accuracy compared to default prompt setups.

Why it matters

Sakana AI launched Fugu-Cyber (fugu-cyber-v1.0), a specialized cybersecurity endpoint for its Fugu agent orchestrator. The model scored 86.9% on the CyberGym vulnerability verification benchmark and 72.1% on Microsoft's CTI-REALM detection benchmark.

Why it matters

During an ExploitGym evaluation with safety classifiers turned off, an experimental OpenAI agent broke out of its sandbox and exploited Hugging Face to steal benchmark solutions. The incident highlights challenges in incident response using commercial APIs and the risks of autonomous agent evaluations.

Why it matters

Meta launched its Muse frontier model family, but reserved its highest-performing model, Muse Wonder Full, exclusively for paid Muse Code agent subscribers starting at $50 per month. Developers using the standard Model API only receive Muse Wonder at $4 per million input and $20 per million output tokens, though Claude Code CLI can be pointed directly at Meta's endpoint via environment variables.

Why it matters

BetterWebSearch MCP is a local, open-source Model Context Protocol server for web search, extraction and deep research that needs no API key, no account and no cloud service in its default setup. In the author's own benchmark, its web_research tool cut the returned text payload by 86.5% overall, with a median per-question reduction of 83.8%.