Oleksandr Kuzmenko, EditorPublished 7 min readAI-assisted · editor-reviewed
TL;DR
- Memory streaming bandwidth from eMMC storage (278 MB/s), not CPU speed, is the primary performance limiter for edge LLMs.
- Mixture of Experts architecture allows running 8B to 16B parameter models inside 1GB of resident memory by touching only necessary weights.
- Headless execution and pruned vocabulary layers free up hundreds of megabytes of RAM on constrained Android and iOS devices.
Key facts
- Hardware Tested
- Xiaomi TV Box S 3rd Gen (1.9GB RAM) and iPad Air 2013 (1GB RAM)
- Flash Read Speed Limit
- 278 MB/s on eMMC flash
- Qwen3.5-0.8B Speed
- 16.8 tokens/second on TV Box
- Moon-Beam-8B-A1B MoE Speed
- 7.2 tokens/second at ~1.1 GB RSS
- Ling-mini-2.0 16B MoE Speed
- 5.1 tokens/second at ~1.13 GB RSS
Breaking the Hardware Memory Wall
Deploying large language models on edge devices typically fails due to out-of-memory errors or severe I/O thrashing. On a Xiaomi TV Box S (3rd Gen) with an Amlogic S7D CPU and 1.9 GB of total RAM, user space processes are limited to approximately 0.96 GB of resident memory, expandable to 1.15 GB in a stripped headless mode. Because eMMC flash throughput peaks at 278 MB/s and random ~1 MB reads incur ~6 ms latencies, streaming dense weights per token enforces a strict mathematical ceiling of 278 / X tokens per second for an X megabyte model.
Mixture of Experts and Custom NEON Kernels
Smolmaxxing bypasses this physical bottleneck by shifting to Mixture of Experts (MoE) architectures and bespoke compute kernels:
- Expert Paging: Using
O_DIRECT, cache-aware routing, and single-layer pre-gated prefetching, the engine streams Q2_K experts on demand, dropping miss rates from 13% to ~2%. - Low-Precision Kernels: 1-bit lookup-table (LUT) kernels and ternary representations packed into 2-bit fields allow dense 8B models like Bonsai-8B to jump from 1.2 tok/s to 5.2 tok/s.
- Pruned Architecture: Trimming vocabulary heads (saving 0.2 to 0.5 GB per model) and adopting headless execution boosts free RAM from ~1,000 MB to ~1,300 MB.
Validated Edge Benchmarks
On the Xiaomi TV Box S, Qwen3.5-0.8B clocks 16.8 tok/s, while Qwen3.5-2B yields 7.7 tok/s. The 128-expert Moon-Beam-8B-A1B achieves 7.2 tok/s, and the 256-expert Ling-mini-2.0 (16.3B parameters) maintains coherent long-form answers at 5.1 tok/s while staying within 1.13 GB RSS. On a jailbroken 2013 iPad Air (Apple A7 CPU, 1 GB RAM, no dotprod), Bonsai-1.7B runs at 24–26 tok/s and Moon-Beam-8B-A1B delivers 3.5 tok/s.
Try it in 2 minutes
git clone https://github.com/hyperionaidev/smolmaxxing && cd smolmaxxingWhen to use
- Deploying local, offline agents or task routers on spare ARM SBCs, routers, or TV boxes.
- Architecting low-cost privacy-first edge devices that cannot afford cloud inference bills.
- Developing specialized memory-paging engines for mobile or embedded AI hardware.
What to do today
- Clone the smolmaxxing repository to review kernel implementations and test measurement logs.
- Profile your edge device storage using sequential and random read benchmarks before attempting large model streaming.
- Evaluate MoE models over dense equivalents when deploying offline inference on memory-constrained hardware.
Step names are taken from What to do today.
Step names are taken from What to do today. Clone the smolmaxxing repository to review kernel implementations and test measurement logs.. Profile your edge device storage using sequential and random read benchmarks before attempting large model streaming.. Evaluate MoE models over dense equivalents when deploying offline inference on memory-constrained hardware.
- Clone the smolmaxxing repository to review kernel implementations and test measurement logs.
- Profile your edge device storage using sequential and random read benchmarks before attempting large model streaming.
- Evaluate MoE models over dense equivalents when deploying offline inference on memory-constrained hardware.
What the community says
It's technically sweet regardless, so don't take this as even a little bit of negative pushback, but I am curious what the utility of these small models is? Are they smart enough to be useful?
in a similar vein, somebody got tinystories LLM running on the nintendo wii, including gpu acceleration [0]. 0: https://github.com/brysontang/iiama2.c
