Llmspeed Simulates Local Large Language Model Token Speeds Before Hardware Purchases
Llmspeed is a web-based performance forecaster that visualizes token-per-second generation and prefill speeds across consumer GPUs, unified APUs, and quantized model weights. It draws on empirical benchmarks from community testing to prevent costly hardware miscalculations.
Why it matters
You can verify whether a quantized coding model will run interactively on target hardware before investing thousands in dedicated GPUs or unified memory laptops.
TL;DR
- 01Prefill latency on large repository contexts is the primary bottleneck for local coding agents, frequently dropping to 100–200 tps on budget memory buses.
- 02Interactive generation speeds of ~20 tps are attainable on legacy hardware once the KV cache is populated.
- 03Simulating hardware specs before upgrading prevents overspending on configurations constrained by unified memory bandwidth.
Key facts
- Data Sources
- r/LocalLLaMA and localscore.ai community benchmarks
- Featured Platform
- AMD Ryzen AI Max+ 395 128GB (Strix Halo)
- Cold-Start Parse Rate
- 100–200 tps on entry-level hardware
- Interactive Throughput
- ~20 tps on warmed KV cache
Forecasting Real-World Local Inference
Local model deployments often fail due to mismatched expectations around token generation latency versus prompt processing overhead. Llmspeed addresses this by aggregating community-submitted hardware figures from r/LocalLLaMA, localscore.ai, and architectural memory bandwidth projections.
The Prefill Bottleneck in Coding Agents
Developers running agentic harnesses such as OpenCode or LM Studio encounter a dual latency profile. While standard generation can sustain interactive rates around 20 tokens per second even on older 8 GB hardware, cold-start prompt evaluation (prefill) for contexts spanning several thousand lines of code often hovers between 100 and 200 tps. Llmspeed highlights how memory bus limitations dictate this delay across platforms like the AMD Ryzen AI Max+ 395 (Strix Halo 128 GB) and multi-GPU setups.
Try it in 2 minutes
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "local-model", "messages": [{"role": "user", "content": "ping"}], "stream": false}' \
-w "\nTotal Time: %{time_total}s | TTFT: %{time_starttransfer}s\n"bash
✓ When to use
- Before purchasing unified memory machines (e.g. Strix Halo, Mac Studio) or consumer GPUs for local vibe-coding workflows.
- When selecting between different GGUF quantization levels to guarantee interactive coding agent speeds.
✕ When NOT to use
- When running models solely via managed cloud APIs (e.g., Claude 3.5 Sonnet or GPT-4o) where hardware limits are abstract.
- When accurate multi-GPU speculative decoding and distributed tensor parallel setups are required.
What to do today
- Input your planned hardware configuration into llmspeed.dev to check estimated prefill and generation speeds.
- Benchmark your current local setup using curl against your local OpenAI-compatible endpoint to establish a baseline time-to-first-token.
- Adjust model quantization to fit within target VRAM and prevent memory spilling into system RAM during large agent queries.
What the community says
“20 tps generation is okay and letting it chug away on my local kubernetes cluster to troubleshoot is pretty nice. First start is pretty slow though, parsing is around 100-200 tps.”
“At least it was quite amazing to see the difference between CPU-based and even this crappy GPU-based inference when trying out some tiny models in jan.ai.”