Local LLMs
Needle 2: 14MB Edge LLM for Device Function Calling
Cactus Compute released Needle 2, a 45M-parameter, 14MB language model built specifically for structured tool calling on budget hardware under $200. Operating with 2-bit quantization inside a single C++ binary, it delivers low-latency offline execution with zero network dependencies.
August 11, 2026 4 min read
Curated by Oleksandr Kuzmenko, AI Product EngineerUpdated August 11, 2026Sources cited on every story
AI-assisted · editor-reviewedHow we use AI
Impact: High
Why it matters
You can now embed reliable, offline function-calling AI into microcontrollers, IoT devices, and mobile apps with a 14MB RAM footprint.
TL;DR
- 01Fits full function-calling capabilities into a 14MB executable requiring minimal RAM.
- 02Uses byte-level schema grammars to guarantee strict JSON and tool parameter syntax.
- 03Ships as a single C++ binary with self-selecting CPU kernels for ARM, x86, and Wasm.
Key facts
- Model Size
- 14 MB
- Parameter Count
- 45 Million
- Quantization
- 2-bit (Cactus Quants)
- Context Window
- 256-token sliding window
- FLOPs per Token
- 70 MFLOPs
Ultra-Compact Architecture for $200 Edge Hardware Needle 2 targets cheap embedded systems, wearables, and microcontrollers by focusing strictly on function mapping and structured data extraction. The model requires only 14MB of RAM and runs on hardware lacking dedicated GPUs or NPUs. Key architectural choices include: - Simple Attention Network: Uses a 256-token sliding window and Walsh-Hadamard transforms to minimize floating-point arithmetic. - Lossless 2-bit Quantization: Trained end-to-end with Quantization-Aware Training (QAT) across weights, activations, and KV cache. - Byte-Level Grammar Enforcement: Constrains decoding logits to candidate schema syntax, skipping up to 98% of vocabulary projections. ### Local Training and Deployment The deliverable is shipped as a self-contained C++ binary supporting ARM Cortex-M, x86, RISC-V, and WebAssembly SIMD. Developers can fine-tune custom tool vocabularies on a standard Mac or PC in minutes using the provided Python package before bundling into production applications.
Try it in 2 minutes
pip install cactus-needle && needle tune --schema tools.json --data train.jsonl --output custom_needle.binbash
✓ When to use
- Running local offline function calling on budget hardware such as $200 microcontrollers or wearables.
- Enforcing strict schema outputs with zero syntax failure tolerance.
✕ When NOT to use
- Open-ended conversational chat or complex multi-step world reasoning.
- Large context tasks requiring long-term document history beyond 256 tokens.
What to do today
- Download the Needle 2 Python package to benchmark function calling latency locally.
- Compile custom tool schemas into byte-level grammars for deterministic outputs.
- Test offline fallback logic using empty call envelopes for low-confidence queries.
What the community says
“i would assume a model this size would require finetuning tbh. even functiongemma recommends that.”
“It seems to fail. I sent the prompt: “ 5° warmer” And it said: “ setting the temperature to 5°F””
#Needle 2#Cactus Quants#vLLM#FunctionGemma
Sources