Search the publication

Results for “Benchmarks”

Showing 1–12 of 80 · latest 80 stories

Benchmarks

04Agents & MCP

Oct 15, 20262 min read

Google Releases Earth AI Agent Framework with Agent-to-Agent Service and Spatial Benchmarks

Google Research launched a Geospatial Reasoning agent architecture connecting specialized planetary foundation models like WeatherNext and AlphaEarth to Gemini. The release includes an Agent-to-Agent API for custom workflows, support for private GeoJSON datasets, and two open benchmarks.

03Vibe coding workflow

Aug 8, 20262 min read

Simon Willison Benchmarks Codex Desktop Sub-Agents in 3D Game Vibe Coding

Simon Willison tested Codex Desktop using GPT-5.6 Sol Ultra in aggressive sub-agent mode to construct a complete 3D web game from a single prompt. While Codex coordinated sub-agents and generated visual assets using gpt-image-2, it failed to fix visual rendering bugs during screenshot self-inspection.

04Models & research

Jul 9, 20262 min read

OpenAI evaluates coding benchmarks and finds thirty percent of tasks are broken

OpenAI published an analysis on evaluating coding LLMs, discovering that roughly 30% of standard benchmark tasks suffer from flaws such as broken tests or invisible requirements.

05Token & cost optimization

Aug 25, 20262 min read

SemiAnalysis AgentX Benchmarks Real-World Agentic AI Token Consumption and Serving Efficiency

NVIDIA published AgentX benchmark data revealing that production AI agents consume 15x more tokens per request than traditional chat sessions. The open-source benchmark replays interactive Claude Code trajectories to measure long-context prefill and KV-cache performance.

07Agents & MCP

Jul 27, 20262 min read

Intel Benchmarks Agentic AI Fleets with vCPU Density and Latency Metrics

Intel released benchmarking research on agentic AI fleet orchestration using an extended version of Terminal-Bench. The report details how agent density per vCPU and P95 latency dictate real-world agent performance.

08Models & research

Sep 2, 20262 min read

AllenAI BenchMIRT Uses Item Response Theory to Audit LLM Benchmarks

The Allen Institute for AI released BenchMIRT, an open-source framework applying multidimensional Item Response Theory to evaluate what benchmark questions actually measure. Testing on 100 LLMs across 34,000 questions showed that retaining just 10% of high-discrimination questions preserves benchmark accuracy.

10Tools & releases

Sep 24, 20262 min read

Tracking Cost-Optimal LLMs on the Pareto Frontier Using Artificial Analysis Benchmarks

An open-source tracker computes the Pareto efficiency frontier of language models from Artificial Analysis data, fetched daily through the free API by a GitHub Actions cron. A budget lookup table points to the highest-scoring model available at each price band and flags models that a cheaper option matches or beats.

10Models & research

Jul 16, 20262 min read

NVIDIA Nemotron 3 Embed Tops Retrieval Benchmarks

NVIDIA released Nemotron 3 Embed, a family of open embedding models with an 8B flagship that ranks #1 on the RTEB leaderboard, plus optimized 1B variants for production-scale retrieval.

10Creative AI

Jul 11, 20262 min read

StoryScope: Narrative Benchmarks Reveal AI Fiction Structural Weaknesses

A new preprint study used over 50,000 AI-generated stories to identify narrative traits that separate human fiction from AI. AI models tend to over-explain themes and lack temporal complexity, making them detectable by structural analysis.

01Tools & releases

Sep 24, 20262 min read

Alibaba Releases Qwen Image 2.1: Compact 7B Open-Weight Generator Rivals Frontier Models

Alibaba has introduced Qwen Image 2.1, an open-weight image model with a 7B parameter footprint. The company claims it beats Google Nano Banana 2.0, while benchmarks show the compact open-weight contender is competitive with OpenAI and Meta image models.

01Models & research

Aug 29, 20262 min read

Anthropic Demonstrates Automated AI Alignment Researchers Operating at Four Dollars per Hour

Anthropic fellow Chen Yueh-Han published research showing automated alignment researchers can reliably improve model benchmarks. Operating via API inference at $4 per hour, the automated system outperformed experienced human researcher proposals within six hours.

01Token & cost optimization

Aug 26, 20262 min read

Quantization-Aware Healing Recovers 4-Bit LLMs Beyond Full-Precision Performance

Quantization-Aware Healing (QAH) recovers compressed 4-bit models by distilling directly from the original full-scale teacher rather than intermediate checkpoints. Applied to a GPT-OSS 120B model compressed to 60B in MXFP4, it outperforms its own 16-bit bfloat16 source on 7 out of 9 benchmarks.