Oleksandr Kuzmenko, EditorPublished 6 min readAI-assisted · editor-reviewed
TL;DR
- Implements full Llama 3.2 1B inference directly in C++17 and CUDA with memory-mapped weights.
- Features a block-table paged KV cache with 16-token pages and FP32 online-softmax decode attention.
- Includes over 80 pages of handwritten architectural notes detailing LLM inference from first principles.
Architecture from Raw Weights to Logits
The samir-vllm repository provides a bare-metal implementation of Llama 3.2 1B Instruct execution. It avoids framework overhead by loading raw BF16 weights directly from model.safetensors via memory mapping (mmap), mapping pointers to all 16 transformer layers without intermediate allocations. It incorporates a custom Unicode BPE tokenizer and custom CUDA kernels for embedding gathering, FP32 RMSNorm reduction, and SwiGLU projections.
Paged Attention and Continuous Scheduling
Rather than allocating static contiguous tensors for long contexts, the engine implements a block-table paged KV cache:
- Physical Page Pool: Defaults to 128 MiB (configurable via
--kv-cache-mib), where each physical block stores 16 tokens of BF16 keys and values. - Online Softmax Attention: During decode, query heads stream sequence pages directly, computing weighted sums in FP32 without materializing the full attention matrix.
- Continuous Batching Scheduler: Admits queued prompts via an admission credit model:
prompt_tokens + allowed_new_tokens - 1. Requests are handled FIFO; if a prompt exceeds available credits, it waits, preventing out-of-memory generation aborts.
Companion Study Notes
The repository includes over 80 pages of handwritten notes and diagrams titled *Inference from Scratch*, covering prefill vs. decode phase dynamics, RoPE rotations, and memory allocation mechanics without AI-generated text.