Local LLMsImpact: High
Oleksandr Kuzmenko, EditorPublished 4 min readAI-assisted · editor-reviewed
TL;DR
- Runs full 27B reasoning and multimodal computer use within 5.9GB of system memory.
- Retains 98.2% baseline accuracy across BigCodeBench, LiveCodeBench v6, and HumanEval+.
- Delivers 143 tokens per second on an RTX 5090 and 46.8 tokens per second on Apple M5 Max.
Key facts
- Model Size
- 5.9GB
- Effective Bits
- 1.76 bits/weight
- Context Window
- 262K tokens
- Benchmark Retention
- 98.2% (self-reported)
- Throughput (RTX 5090)
- 143 tokens/sec
- License
- Apache 2.0
Source: Key facts — Model Size, Effective Bits, Context Window, Throughput (RTX 5090)
Source: Key facts. Model Size: 5.9GB. Effective Bits: 1.76 bits/weight. Context Window: 262K tokens. Throughput (RTX 5090): 143 tokens/sec. Each figure is copied from the story.
- Model Size5.9GB
- Effective Bits1.76 bits/weight
- Context Window262K tokens
- Throughput (RTX 5090)143 tokens/sec
Extreme Quantization Without Agent Drift Ternary Bonsai 2 27B, built on top of Qwen3.8 27B, reduces model size by over 9x to 5.9GB using 1.76 effective bits per weight. While previous sub-2-bit experiments failed on multi-turn coding benchmarks due to cascading logic errors, Bonsai 2 scores 83.9 on aggregate benchmarks, retaining 98.2% of its full-precision baseline.
Hardware Throughput and Agent Integration The model achieves 143 tokens/s on desktop NVIDIA RTX 5090 cards and 46.8 tokens/s on Apple M5 Max hardware via native MLX kernels. With an energy draw of 0.714 mWh per token on an RTX 4090, it operates at 40% lower energy consumption than a standard full-precision 8B parameter model. Prism ML verified compatibility with Cline agent loops and multimodal computer-use tasks across a 262K context window under Apache 2.0.
Try it in 2 minutes
bash
git clone https://github.com/prismml/bonsai && cd bonsai && pip install -e . && python -m bonsai.serve --model ternary-bonsai-2-27b --context-window 262144When to use
- Running offline agentic coding pipelines with strict privacy requirements.
- Hosting multi-turn tool-calling loops on consumer laptops or single GPU workstations.
- Budgeting long-running background reasoning where cloud API costs are prohibitive.
What to do today
- Download the model weights from the official Apache 2.0 repository.
- Benchmark local inference speed using the custom MLX kernel on Apple Silicon or CUDA on RTX GPUs.
- Hook the local endpoint into Cline or Cursor to test multi-turn agentic editing without cloud fees.
Step names are taken from What to do today.
Step names are taken from What to do today. Download the model weights from the official Apache 2.0 repository.. Benchmark local inference speed using the custom MLX kernel on Apple Silicon or CUDA on RTX GPUs.. Hook the local endpoint into Cline or Cursor to test multi-turn agentic editing without cloud fees.
- Download the model weights from the official Apache 2.0 repository.
- Benchmark local inference speed using the custom MLX kernel on Apple Silicon or CUDA on RTX GPUs.
- Hook the local endpoint into Cline or Cursor to test multi-turn agentic editing without cloud fees.