Oleksandr Kuzmenko, EditorPublished 5 min readAI-assisted · editor-reviewed
TL;DR
- The repository demonstrates how state-of-the-art architectures combine GDN linear recurrence and MLA to eliminate KV-cache bottlenecks.
- Extreme MoE scaling has shifted toward wide expert counts (1,224 experts) with granular top-k selection (top-48 plus shared).
- Direct inference is computationally unfeasible without enterprise clusters running dozens of Trainium2 or Blackwell nodes.
Key facts
- Total Reported Parameters
- ~14T (6.85T tensor parameters in index)
- Total Shard Size
- 7.72 TB across 772 safetensors shards
- Layer Count
- 114 layers
- MoE Configuration
- 1,224 experts (top-48 routed + 1 shared)
- Primary Precision
- NVFP4 (E2M1 payloads with E4M3 scales)
Source: Key facts — Layer Count
Source: Key facts. Layer Count: 114 layers. Each figure is copied from the story.
- Layer Count114 layers
Architecture Breakdown: GDN Meets Multi-Head Latent Attention
The burner22/numbat repository publishes a reference implementation in configuration_numbat.py and modeling_numbat.py designed to load 772 safetensors files totaling 7,720,640,941,879 bytes (~7.72 TB). According to the model card, the architecture consists of 114 layers configured as a hybrid stack:
- Attention Layers: Alternating Gated DeltaNet (GDN) linear recurrent state-space modules and Multi-head Latent Attention (MLA) projections to compress KV-cache memory footprints.
- Sparse Routing: A Mixture-of-Experts feed-forward network containing 1,224 routed experts, activating top-48 experts per token alongside 1 permanently activated shared expert.
- Speculative Heads: Built-in Multi-Token Prediction (
NumbatForCausalLM.draft) executed alongside the main model trunk (NumbatForCausalLM.forward).
Tensor Encoding and Hardware Feasibility
The parameter tensors are distributed across 6.83 trillion uint8 parameters backing NVFP4 quantized weights, with 16.85 billion BF16 parameters reserved for normalizations and embedding matrices. Scales follow an E4M3 block format decoding E2M1 payloads. Running or fine-tuning this checkpoint is out of reach for individual practitioners; the author notes that evaluation runs required roughly 85 AWS Trainium2 chips just to retain the NVFP4 tensor allocations in memory.
Try it in 2 minutes
from safetensors import safe_open
# Inspect shard header metadata without downloading full multi-terabyte tensors
with safe_open('model-00001-of-00772.safetensors', framework='pt') as f:
print(f"Keys in shard 1: {len(f.keys())}")When to use
- Read modeling_numbat.py when designing custom PyTorch architectures that combine state-space recurrence with attention.
- Reference the shard metadata structure when writing distributed model loaders for massive MoE checkpoints.
When NOT to use
- Do not download the 7.72 TB weight shards for local inference or fine-tuning without dedicated datacenter infrastructure.
- Do not deploy unverified third-party checkpoints into production systems or corporate environments.
What to do today
- Clone modeling_numbat.py to inspect the PyTorch forward pass implementation of hybrid GDN and MLA blocks.
- Review the E4M3 scale decoding routines in safetensors to understand production NVFP4 dequantization mechanics.
Step names are taken from What to do today.
Step names are taken from What to do today. Clone modeling_numbat.py to inspect the PyTorch forward pass implementation of hybrid GDN and MLA blocks.. Review the E4M3 scale decoding routines in safetensors to understand production NVFP4 dequantization mechanics.
- Clone modeling_numbat.py to inspect the PyTorch forward pass implementation of hybrid GDN and MLA blocks.
- Review the E4M3 scale decoding routines in safetensors to understand production NVFP4 dequantization mechanics.
What the community says
Obviously fake. Come on. This isn’t like leaking llama… even if it were real you would need several racks of b200s to even run this at all coupled with nvlink.
A day will come when such a leak will happen - thats almost a certainty. But it's not today.