Liquid AI Releases LFM2.5 Q4_0 GGUF Models Using Quantization-Aware Distillation
Liquid AI has released Q4_0 GGUF 4-bit checkpoints for its LFM2.5 model series, trained using Quantization-Aware Distillation. These models recover up to 97% of accuracy lost to traditional quantization while maintaining native 4-bit speeds.
Impact: Medium
Why it matters
Developers can now run highly accurate, lightweight LFM2.5 models locally with minimal memory footprint and zero speed degradation.
TL;DR
- 01Quantization-Aware Distillation (QAD) recovers up to 97.4% of original model performance lost to 4-bit quantization.
- 02Small models (230M and 350M) match Q5_K_M precision with up to 33% higher decoding speeds.
- 03These GGUF checkpoints are optimized for edge hardware, including macOS GPU and Arm CPUs.
Key facts
- Accuracy Recovered
- Up to 97.4% of BF16 baseline
- Supported Models
- LFM2.5 (230M, 350M, 1.2B, 2.6B)
- Format
- Q4_0 GGUF
- Throughput Gain (230M/350M)
- 4% to 33% over Q5_K_M
High-Precision Distillation for the Edge
Liquid AI has introduced Quantization-Aware Distillation (QAD) to bridge the gap between model size, execution speed, and reasoning capabilities. By distilling knowledge from a high-precision teacher model into a quantized student, the new Q4_0 GGUF checkpoints recover 97.1%, 96.5%, 97.4%, and 96.6% of their respective BF16 baseline performance across LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B.
Performance and Throughput Benchmarks
The updated checkpoints deliver native Q4_0 execution speeds and memory footprints while matching the accuracy of heavier quantization formats:
- LFM2.5-230M and 350M: Match
Q5_K_Mquality with 4% to 33% higher decode throughput. - LFM2.5-1.2B and 2.6B: Match
Q4_K_Mquality with 3% to 14% higher decode throughput. - Compatibility: Matches Unsloth's
UD-Q4_K_XLwhere applicable (for 230M and 1.2B models).
Deployment Targets
Developers can run these models locally using llama.cpp or any GGUF-compatible runtime. Hardware compatibility has been verified across GPU targets (MacBook Pro, NucBox EVO-X2) and Arm CPU targets (Samsung Galaxy S26 Ultra, Raspberry Pi 5).
Try it in 2 minutes
./llama-cli -m LFM2.5-1.2B-Instruct-QAD-Q4_0.gguf -p "Write a python function to merge two sorted arrays."bash
✓ When to use
- When deploying small, highly efficient LLMs on edge devices like Raspberry Pi 5 or mobile phones.
- When you need the memory footprint of a 4-bit model but require the accuracy of 5-bit or original precision.
- For low-latency, local agentic workflows or tool-use scenarios.
✕ When NOT to use
- When your task demands extremely large parametric knowledge bases that only 70B+ models can provide.
- If your deployment runtime does not support the GGUF Q4_0 standard format.
What to do today
- Download the new QAD Q4_0 GGUF models from Hugging Face.
- Benchmark local inference throughput on your target edge hardware using llama.cpp.
Sources