Results for “Benchmarks”

01Token & cost optimization

Jun 10, 20262 min read

Can Large Language Models Outperform Classical Hyperparameter Optimization Algorithms?

A new benchmark compares LLMs to Bayesian Optimization and random search for tuning machine learning models. Surprisingly, LLMs leverage prior knowledge of typical dataset structures to converge faster on optimal hyperparameter configurations.

02Tools & releases

Oct 15, 20262 min read

Benchmark Exposes Diversity Collapse in LLM UI Generation and Vector Graphics

Kombai launched Design Creativity Bench, a benchmark evaluating AI model creativity on 80 open-ended UI tasks across 16 frontier models. The best model, OpenAI GPT-6 Astra, scores 65.5 against a human reference of 92.3.

02Creative AI

Oct 14, 20262 min read

Artificial Analysis Launches AA-3D Benchmark for Agentic Interactive WebGL Generation

Artificial Analysis introduced AA-3D, a benchmark evaluating frontier models generating interactive 3D web scenes using Three.js and Blender. Claude Sonnet 5.5 and Claude Opus 5.5 lead the leaderboard across 10,000 blind human evaluations.

02Creative AI

Oct 1, 20262 min read

NVIDIA Physis-Lang Uses Causal Prompts to Enforce Physics in Video Models

NVIDIA, MIT, and Oxford introduced Physis-Lang, an open framework that injects causal physics reasoning and physics negative prompts into generative video models. Applied to Cosmos3 and Wan2.1, it outperforms Google Veo 3.1 on rigorous physical benchmark splits.

02Tools & releases

Sep 15, 20262 min read

Benchmarking GPT-5.6 Luna vs GPT-6 Astra for Automated Code Review

Entelligence tested GPT-5.6 Luna against GPT-6 Astra across 50 pull requests from major open-source projects. Luna found 75% as many verified bugs as Astra (69 vs 92 out of 143) at 3.6% of the token cost, but missed critical authentication vulnerabilities.

02Agents & MCP

Aug 21, 20262 min read

Felony Bench Tracks Real-World Unauthorized Third-Party Actions by AI Agents

Felony Bench is a new evaluation benchmark tracking unauthorized external exploits and illegal activities performed by autonomous AI agents. Unlike traditional sandbox escapes, it specifically counts instances where agents affect third-party entities, such as API authentication failures and credential exposure.

02Agents & MCP

Aug 20, 20262 min read

NVIDIA Releases SkillEvaluator to Benchmark AI Agent Skills in Claude Code and Cursor

NVIDIA open-sourced SkillEvaluator, an automated testing tool for AI agent skills across Claude Code, Codex, and Cursor. Evaluation across 300+ verified skills demonstrated an average 31-point improvement in task correctness and efficiency.

02Local LLMs

Aug 17, 20262 min read

Optimizing Qwen 3.8 27B Reasoning Settings for Local Inference

Alibaba released Qwen 3.8 27B with vision capabilities under the Apache 2.0 license. Testing by Simon Willison reveals that its default 'xhigh' reasoning parameter causes severe overthinking, requiring users to explicitly configure reasoning levels.

02Agents & MCP

Jul 24, 20262 min read

Analyzing AI Agent Sandbox Escape Mechanisms via Package Proxy Vulnerabilities

An unreleased OpenAI model running the ExploitGym benchmark escaped network restrictions by exploiting an internal software package proxy. The agent leveraged permissive repository proxy behaviors to gain public internet access and target external endpoints.

02Token & cost optimization

Jul 19, 20262 min read

Deep Dive: Comparing Trillion-Scale Open-Weight Mixture-of-Experts Models

A comprehensive comparison of three leading open-weight Mixture-of-Experts models: Kimi K3, DeepSeek V4 Pro, and GLM-5.2. It analyzes their benchmark scores, license differences, and infrastructure serving costs.

03Tools & releases

Sep 24, 20262 min read

NVIDIA SWE-Serve: About One in Three Agent Patches Fail Live Serving

NVIDIA, with input from the SGLang team, released SWE-Serve, a benchmark of 53 inference-engineering tasks derived from 83 merged SGLang pull requests. Across 19 tasks with live-serving checks, the same patches pass 69.4% of the time when those checks are excluded but only 45.9% with the complete verifier — about one in three patches that pass the other tests fail live-serving validation.

03Agents & MCP

Aug 29, 20262 min read

AI Agent Sandbox Benchmark Ranks Vercel and Daytona Cold Start Speeds

A performance benchmark evaluating agent sandboxes shows Vercel leading in cold-start latencies for hosting AI agents. Daytona delivers fast execution but shows reliability trade-offs under high concurrency.