Evaluating Twelve Frontier Models in a Multi-Run App Build-Off
A comprehensive testing suite evaluated 12 frontier models across five attempts each to build four complex visual applications, revealing that GPT models lead in 3D rendering while Claude Fable 5 dominates aesthetic and layout consistency. It offers realistic data on run-to-run variance, execution costs, and failure rates.
Impact: Medium
Why it matters
You can review raw builds and cost analyses to select the most reliable model for specific vibe-coding tasks like 3D physics or styled interfaces.
TL;DR
- 01Multi-run testing reveals high variance in code generation; never trust a single prompt for complex visual applications.
- 02Claude Fable 5 remains the premier model for sophisticated UI layout and CSS styling consistency.
- 03GPT-5.6 Sol is best suited for complex algorithmic logic and 3D calculations like raycasting.
- 04Routing simple, boilerplate tasks like the Game of Life to Qwen 3.7 Plus can slash prompt costs by up to 95%.
Key facts
- Raycaster Sol Cost
- $1.35 (5 runs)
- Rubik's Fable 5 Score
- 5/5 Clean Solves
The Run-to-Run Variance Problem
One-shot benchmarks often hide how volatile LLM outputs can be. In this multi-run evaluation, each model was tasked with building the same applications five times to assess consistency. The results show that relying on a single generation is highly risky for production-grade vibe-coding. Even the most expensive models produced entirely broken artifacts on certain runs. For example, GPT-5.6 Sol rendered an all-black Rubik's cube in one of its attempts, while Muse Spark 1.1 had three out of five attempts fail on the raycaster despite rendering spectacular results when it succeeded.
Model Performance and Cost Analysis
The benchmark highlights distinct capabilities across the frontier lineup:
- First-Person Raycaster: GPT-5.6 Sol achieved a perfect 5/5 playable score, costing $1.35 across five runs with a 120-second average time. Grok 4.5 also hit 5/5 playability at a fraction of the cost ($0.27).
- 3D Rubik's Cube: Claude Fable 5 was the only model to achieve 5/5 flawless solves with perfect animations, costing $2.03. In contrast, GPT-5.6 Luna failed to deliver a single clean solve (0/5) because scrambling broke the visual state immediately.
- Complex Calculator: Claude Fable 5 and Opus 4.8 both hit 5/5. GPT-5.6 Sol over-styled its UI, which led to broken element alignments on some runs.
Choosing Open Weights for Simple Layouts
For well-documented tasks like Conway's Game of Life, open-weights models like Qwen 3.7 Plus and GLM-5.2 executed perfectly at trivial costs. Qwen 3.7 Plus completed calculator tasks for just $0.04 across five runs in 12 seconds, indicating that routing simple ui tasks to smaller, open-weights endpoints can radically optimize developer spending.
✓ When to use
- When selecting which frontier model to utilize for autonomous frontend development and rapid prototyping.
- When budgeting prompt costs for AI-native agent workflows that build visual layouts.
✕ When NOT to use
- Not suited as a definitive scientific evaluation of model architecture.
- Not applicable for evaluating backend server systems that do not involve a user interface.
What to do today
- Run complex visual code generations at least three times to evaluate layout and state stability.
- Use Grok 4.5 as a cheaper fallback for 3D physics code if GPT-5.6 is too expensive for iterative testing.
What the community says
“One-shot benchmarks are great for me as a solo creator, since they slightly correlate to whether the better frontier models (Opus and Fable for me) make better decisions about things I didn't spec”
“I gave up on Grok. It constantly ignores explicit instructions (e.g. do NOT remove existing comments) and it's not nearly as intuitive in knowing the right questions to ask”
“I imagine one could one-shot a basic app and then feed feature requests one by one, sounds like an obvious way to benchmark architecture/maintainability”
Sources