OpenAI Jalapeño Custom ASIC Benchmarked: 1,400 Tokens Per Second on Open Models
OpenAI's self-designed Jalapeño inference chip achieved over 700 tokens per second on DeepSeek R1 and 1,400 tokens per second on GPT-OSS during initial laboratory benchmarks. Built with HBM4 memory, it outperforms Nvidia Blackwell in token output per megawatt.
Impact: Medium
Why it matters
Hardware-software co-design is rapidly driving down the latency floor for local and open reasoning models, enabling ultra-fast real-time agent execution.
TL;DR
- 01OpenAI's Jalapeño ASIC hits 700 to 1,400 tokens/sec on open reasoning models in lab settings.
- 02Equipped with HBM4 memory, Jalapeño prioritizes throughput per megawatt to combat datacenter power constraints.
- 03Benchmarked workloads rely on single-turn 8k context; multi-turn long-context performance on AgentX is pending.
Key facts
- Architecture
- Custom inference ASIC with HBM4 memory
- DeepSeek R1 Throughput
- >700 tokens/sec/user (Concurrency 1, self-reported)
- GPT-OSS / Kimi K2.5 Throughput
- ~1,400 tokens/sec/user (self-reported)
- Development Cycle
- 16 months from team hiring to tape-out
- Benchmark Suite
- InferenceX v3 (8k1k single-turn evaluation)
Hardware Architecture and Timeline
OpenAI collaborated with Broadcom starting in mid-2024 to design Jalapeño, going from blank slate to manufacturing tape-out in approximately 16 months. Designed primarily for LLM inference, the chip integrates high-bandwidth memory (HBM4) to keep pace with modern frontier model weights and KV-cache demands.
Inference Performance Metrics
In laboratory runs on the InferenceX evaluation suite, Jalapeño reached non-speculative single-token prediction speeds across multiple open architectures:
- DeepSeek R1: Exceeded
700 tokens/sec/userat concurrency 1. - GPT-OSS and Kimi-K2.5: Reached
1,400 tokens/sec/useroutput throughput. - GSM8k Evals: Maintained accuracy on par with Nvidia standard runs.
Power Efficiency and Caveats
Data centers are strictly power-constrained; token generation per megawatt (MW) directly governs revenue. Jalapeño's throughput per MW surpasses Nvidia Blackwell (GB200) and early Vera Rubin results in single-token prediction mode. However, these figures reflect 8k context input and 1k output runs. Real-world agentic workloads involving multi-turn long-context caching (tested via AgentX) may expose different routing and memory bottlenecks.
✓ When to use
- Evaluating latency expectations and cost curves for next-generation frontier model hosting.
- Designing high-frequency agent interaction loops that benefit from 1,000+ tokens/sec output rates.
What to do today
- Monitor upcoming independent AgentX benchmark results for long-context multi-turn latency numbers on custom ASICs.
- Audit application streaming parsers to ensure they can process high-velocity token streams exceeding 1,000 tokens/sec without UI freeze.
What the community says
“... better than Blackwell in this specific case; which can also lead to OpenAI creating models that will only work on their own chips; vendor lock-in.”
“Well, with proper compaction algorithm (like in Codex), it could indeed be useful for majority of tasks even in the future.”
Sources