Tutorials & guides
TokenTown Visualizes Transformer Mechanics and Key-Value Caching in Interactive City Model
TokenTown is an interactive browser visualizer that maps each stage of a transformer language model to an isometric city district, simulating attention, KV caching, prefill, and decode phases.
July 27, 2026 3 min read
AI-assisted · editor-reviewedHow we use AI
Why it matters
Explore TokenTown to build an intuitive mental model of prompt prefilling, token decoding, and KV cache memory dynamics.
TL;DR
- 01TokenTown simulates transformer architecture in a live browser-based isometric city.
- 02Demonstrates real differences between parallel prefill and sequential token decoding.
- 03Visualizes KV cache growth and multi-head scaled dot-product attention in real time.
Key facts
- Model Size
- 12 dimensions, 2 heads, 2-12 layers
- Interactivity Controls
- Space (Play/Pause), S (Step), R (Reset), Speed slider (0.4x-8x)
Inside the TokenTown Architecture
The visualizer scales down large language model parameters to run live in JavaScript:
- Model Dimensions: 12 dimensions, 2 attention heads, 2–12 configurable layers
- Vocabulary: A few hundred words
- Mechanics Rendered: Tokenizer split, embedding lookup, sinusoidal positional encoding, LayerNorm, multi-head attention with causal masking over a live KV cache, GELU feed-forward, and top-p sampling.
Prefill vs Decode Execution
The simulation directly highlights execution differences:
- Prefill Phase: Processes the entire input prompt simultaneously through attention layers.
- Decode Phase: Generates tokens sequentially, demonstrating why autoregressive generation requires persistent KV cache storage.
✓ When to use
- When explaining LLM architecture, KV cache memory overhead, or prefill vs decode stages to engineering teams.
- When building visual intuition for how transformer tokens flow through residual attention layers.
What to do today
- Visit laurentiugabriel.github.io/token-town to run the interactive tour.
- Use Space to pause and S to step through transformer layer dynamics step-by-step.
What the community says
“Looks like the PG world website we've seen on the frontpage. Did you get some inspiration from that project?”
#TokenTown#Transformer
Sources