Benchmarking GPT-5.6 Luna vs GPT-6 Astra for Automated Code Review
Entelligence tested GPT-5.6 Luna against GPT-6 Astra across 50 pull requests from major open-source projects. Luna found 75% as many verified bugs as Astra (69 vs 92 out of 143) at 3.6% of the token cost, but missed critical authentication vulnerabilities.
Impact: Medium
Why it matters
Frontier models can be prohibitively expensive for scanning every CI/CD pull request. Understanding where budget models hold up—and where their security blindspots lie—enables cost-effective multi-model review setups.
TL;DR
- 01Luna found 75% as many verified bugs as Astra (69 vs 92 of 143) at 3.6% of the cost.
- 02Luna suffers a 25.8% verification failure rate compared to Astra's 4.2%.
- 03Luna missed the majority of security defects, catching only 9 of 24 compared to Astra's 19.
- 04Running both models on every PR captured 117 of 143 bugs (82%) for $5.86 total.
Key facts
- GPT-5.6 Luna Pricing
- $0.20 input / $1.20 output per 1M tokens
- GPT-6 Astra Pricing
- $10 input / $50 output per 1M tokens
- Average Cost per Review
- $0.0041 (Luna) vs $0.113 (Astra)
- Average Review Latency
- 23s (Luna) vs 36s (Astra)
- Security Bug Detection
- 9/24 caught by Luna vs 19/24 by Astra
- Verification Failure Rate
- 24/93 (25.8%) for Luna vs 4/96 (4.2%) for Astra
Cost and Latency Tradeoffs
GPT-5.6 Luna costs $0.20 per million input tokens and $1.20 per million output tokens, whereas GPT-6 Astra costs $10 input and $50 output. Over 50 benchmark PRs containing introduced defects, Luna cost $0.20 total ($0.0041 per PR) versus $5.66 total ($0.113 per PR) for Astra. Luna executed in 23 seconds per review versus Astra's 36 seconds.
Detection Rates and the Security Blindspot
Luna found 69 verified bugs (75% as many as Astra), while Astra caught 92 out of 143 total benchmark bugs identified. However, Luna suffered a higher noise ratio: 24 of its 93 reported issues were rejected under dual-judge verification, compared to 4 out of 96 for Astra. The performance gap diverged sharply by bug category:
- Data and Logic: Luna found 39; Astra found 47.
- Concurrency: Luna found 10; Astra found 13.
- Security: Luna found 9 of 24; Astra found 19 of 24.
On the Keycloak benchmark, where PRs modified authentication and permission logic, Luna managed only 6 verified bugs against Astra's 14, with a 50% false-positive rate. Luna failed to detect federated recovery codes that were never marked as used and global view permissions overriding individual client denials.
Multi-Model Review Gains
Interestingly, 25 verified bugs were discovered solely by Luna, including concurrency issues in Sentry and notification logic regressions in Discourse. Running both models on every pull request would have found 117 of the 143 verified bugs (82%) for $5.86 in total ($0.20 for Luna plus $5.66 for Astra), adding 25 verified bugs beyond Astra alone.
✓ When to use
- For high-volume, low-budget triage of general data and logic bugs on everyday diffs.
✕ When NOT to use
- When reviewing security-critical authorization, token validation, or cryptographic routines without human or frontier oversight.
What to do today
- Avoid using budget models alone on authentication, IAM, and permissions code.
- Consider ensembling models to capture complementary bug findings across logic and concurrency.
Sources