Fine-Tune a 350M Model for Structured Outputs Using GRPO
Hugging Face published a lightweight recipe to train small models for strict schema adherence using Group Relative Policy Optimization (GRPO) in TRL. With just 100 steps and 500 samples on a free GPU, output adherence rose from 22.6% to 29.7% on IFStruct.
Impact: Medium
Why it matters
You can deploy fast, local sub-1B models for agent tool-calling pipelines without paying proprietary API costs or suffering malformed JSON errors.
TL;DR
- 01100 GRPO steps improve 350M parameter model schema compliance from 22.6% to 29.7% on IFStruct.
- 02LoRA on hybrid architecture modules trains only 6M parameters (1.66%), fitting comfortably on 16GB free-tier GPUs.
- 03Exported BF16 GGUF weights run directly in local pipelines through standard llama.cpp servers.
Key facts
- Base IFStruct Score
- 22.6%
- Tuned IFStruct Score
- 29.7%
- Training Steps
- 100
- Dataset Samples
- 500
- Trained Parameters
- 6M (1.66%)
- GPU Memory Footprint
- 16 GB
Training Pipeline and LoRA Setup
The implementation pairs Hugging Face's TRL library with Liquid AI's LFM2.5-350M. Because LFM2.5 utilizes a hybrid attention and convolution architecture, LoRA targets projection modules q_proj, k_proj, v_proj, out_proj, and in_proj. This trains approximately 6 million parameters, which represents only 1.66% of the model weights.
Multi-Component Reward Modeling
Rather than relying on generic reinforcement learning penalties, the pipeline defines three explicit reward functions on a [0, 1] scale:
json_format_reward(weight1.0): Grants full credit for requested formatting (fenced vs raw JSON),0.2for valid but non-conforming formats, and0.0for invalid outputs.field_count_reward(weight0.5): Evaluates top-level field counts, degrading linearly for missing or extraneous keys.schema_validation_reward(weight2.0): Validates keys and constraints against target JSON schemas, gating credit on required fields.
Execution and Local Deployment
The training configuration specifies num_generations=8, gradient_accumulation_steps=8, max_completion_length=1024, temperature=1.1, and a KL penalty beta=0.01. Training completes in 100 steps on a 16 GB GPU. The merged weights are subsequently converted to a BF16 GGUF model and served locally on llama-server with --alias lfm25-350m-grpo-structured-output, raising baseline IFStruct performance from 22.6% to 29.7%.
Try it in 2 minutes
git clone --depth 1 https://github.com/ggml-org/llama.cpp
python llama.cpp/convert_hf_to_gguf.py --outfile ./models/lfm25-350m-grpo-bf16.gguf ./outputs/lfm25-350m-nemotron-schema-grpo-mergedbash
✓ When to use
- When running extraction agents on local hardware or edge environments with strict memory constraints.
- When building deterministic JSON tool-calling pipelines powered by local llama.cpp endpoints.
✕ When NOT to use
- When your downstream service requires multi-turn long-document reasoning and code synthesis beyond short schema parsing.
- When hosted frontier models with native constrained decoding fall well within your latency and cost budgets.
What to do today
- Clone the Liquid4All IFStruct repository to evaluate your local baseline schema compliance.
- Configure TRL GRPO training parameters with 8 generations per prompt and a 16GB GPU budget.
- Convert merged LoRA checkpoints to BF16 GGUF format using llama.cpp conversion scripts.
Sources