Tutorials & guides
TokenTown Visualizes Transformer Mechanics and Key-Value Caching in Interactive City Model
July 27, 2026 3 min read
AI-assisted · editor-reviewedHow we use AI
TokenTown is an interactive browser visualizer that maps each stage of a transformer language model to an isometric city district, simulating attention, KV caching, prefill, and decode phases.
Why it matters
Explore TokenTown to build an intuitive mental model of prompt prefilling, token decoding, and KV cache memory dynamics.
TL;DR
- 01TokenTown simulates transformer architecture in a live browser-based isometric city.
- 02Demonstrates real differences between parallel prefill and sequential token decoding.
- 03Visualizes KV cache growth and multi-head scaled dot-product attention in real time.
Key facts
- Model Size
- 12 dimensions, 2 heads, 2-12 layers
- Interactivity Controls
- Space (Play/Pause), S (Step), R (Reset), Speed slider (0.4x-8x)
Inside the TokenTown Architecture
The visualizer scales down large language model parameters to run live in JavaScript:
- Model Dimensions: 12 dimensions, 2 attention heads, 2–12 configurable layers
- Vocabulary: A few hundred words
- Mechanics Rendered: Tokenizer split, embedding lookup, sinusoidal positional encoding, LayerNorm, multi-head attention with causal masking over a live KV cache, GELU feed-forward, and top-p sampling.
Prefill vs Decode Execution
The simulation directly highlights execution differences:
- Prefill Phase: Processes the entire input prompt simultaneously through attention layers.
- Decode Phase: Generates tokens sequentially, demonstrating why autoregressive generation requires persistent KV cache storage.
✓ When to use
- When explaining LLM architecture, KV cache memory overhead, or prefill vs decode stages to engineering teams.
- When building visual intuition for how transformer tokens flow through residual attention layers.
What to do today
- Visit laurentiugabriel.github.io/token-town to run the interactive tour.
- Use Space to pause and S to step through transformer layer dynamics step-by-step.
What the community says
“Looks like the PG world website we've seen on the frontpage. Did you get some inspiration from that project?”
#TokenTown#Transformer
Sources