Quantization-Aware Healing Recovers 4-Bit LLMs Beyond Full-Precision Performance
Quantization-Aware Healing (QAH) recovers compressed 4-bit models by distilling directly from the original full-scale teacher rather than intermediate checkpoints. Applied to a GPT-OSS 120B model compressed to 60B in MXFP4, it outperforms its own 16-bit bfloat16 source on 7 out of 9 benchmarks.
Impact: High
Why it matters
Engineers can now deploy half-sized, 4-bit models that deliver higher accuracy and lower latency than their 16-bit checkpoints.
TL;DR
- 01QAH distills 4-bit student models directly from uncompressed full-scale teachers using KL-divergence.
- 02QAH models beat their 16-bit bfloat16 checkpoints on key math (+5.6 AIME 2025) and reasoning benchmarks.
- 03QAH converges 7x faster than traditional QAT (100 steps vs 700) and prevents catastrophic accuracy drift.
Key facts
- GPT-OSS 60B MXFP4 LiveCodeBench
- 66.5 (vs 16-bit 66.0)
- AIME 2025 Delta vs 16-bit
- +5.6 points
- AA-LCR Delta vs 16-bit
- +7.4 points
- Convergence Steps (QAH vs QAT)
- 100 steps vs 700 steps
Quantization Ceiling Broken
Traditional LLM quantization typically degrades reasoning performance. When applying Quantization-Aware Healing (QAH) to a GPT-OSS 120B model compressed down to 60B parameters in MXFP4, the resulting model wins against its own 16-bit bfloat16 checkpoint on 7 out of 9 standard benchmarks.
Benchmark Results and Training Stability
- Long-Context Reasoning (AA-LCR): +7.4 point gain over
bfloat16. - Mathematics (AIME 2025): +5.6 point gain over
bfloat16. - Code Generation (LiveCodeBench): Scores
66.5, outperforming the full-size 120B teacher at66.0. - GPQA Diamond: Scores
67.4compared to teacher's69.0.
In stability comparisons on a GPT-OSS 9B architecture in MXFP4, QAH achieved peak accuracy (54.9) in 100 steps versus QAT taking 700 steps (54.6). Beyond 1,000 steps, QAT degraded by almost 19 points due to cross-entropy overfitting, while QAH remained within 2 points of its peak.
Try it in 2 minutes
# Example concept for chunked KL-divergence loss calculation
import torch
import torch.nn.functional as F
def chunked_kl_loss(student_logits, teacher_logits, chunk_size=1024):
loss = 0.0
total_tokens = student_logits.size(1)
for i in range(0, total_tokens, chunk_size):
s_chunk = F.log_softmax(student_logits[:, i:i+chunk_size, :], dim=-1)
t_chunk = F.softmax(teacher_logits[:, i:i+chunk_size, :], dim=-1)
loss += F.kl_div(s_chunk, t_chunk, reduction='batchmean')
return loss / (total_tokens / chunk_size)python
✓ When to use
- When quantizing large models (30B+) to 4-bit precision for high-throughput local GPU deployment.
- When standard QAT fine-tuning suffers from instability, overfitting, or performance collapse.
✕ When NOT to use
- When original pre-compression full-precision model logits are unavailable for teacher offline caching.
- For small models below 3B parameters where structural pruning causes unrecoverable capacity loss.
What to do today
- Evaluate QAH KL-divergence distillation pipelines when deploying local 4-bit MoE or dense LLMs.
- Re-evaluate MXFP4 precision options for edge and local agent runtime deployments.
Sources