nanoRL: Minimal 1,800-Line Framework for Reinforcement Learning Training of LLMs
nanoRL implements disaggregated, asynchronous RL training loops for LLMs in 1,800 lines of Python without Ray, DeepSpeed, or TRL. It supports REINFORCE, PPO, GRPO, and RLOO across single CPUs up to GPU clusters.
Impact: Medium
Why it matters
You can hack, debug, or fork a transparent RL post-training pipeline for LLMs without dealing with complex orchestration dependencies.
TL;DR
- 01nanoRL distills complex RL post-training (PPO, GRPO, RLOO) into ~1,800 lines of readable Python.
- 02Decoupling vLLM generation from PyTorch training reduces weight sync transfer sizes to ~170 MB via LoRA.
- 03Recomputing old logprobs on the trainer is necessary to prevent accuracy collapse caused by kernel math differences.
Key facts
- Codebase Size
- ~1,800 lines across 7 Python files
- Supported Algorithms
- REINFORCE, PPO (GAE), GRPO, RLOO
- LoRA Adapter Sync Size
- ~170 MB (vs 16 GB full weights)
- Dependencies
- PyTorch, vLLM, Hugging Face Transformers, PEFT
Core Architecture and Algorithm Zoo
All RL algorithms in nanoRL reduce to computing advantage values in algos.py. The repository spans 7 files including train.py (loop and updates), model.py (vLLM generator and Hugging Face policy wrapper), serve.py (async transport over stdlib HTTP), and tasks.py (prompting and reward functions).
Asynchronous Disaggregated Training
Workers communicate with the trainer over standard HTTP. Generation workers execute vLLM with separate batch sizes optimized for sampling, while the trainer executes micro-batched updates. Weight synchronization transfers only LoRA adapter parameters (~170 MB) rather than full model checkpoints (~16 GB), allowing 200 GRPO steps on 102k sequences in 90 minutes on 8x H100 plus 8x L40S nodes.
Logprob Precision and Mismatch Prevention
Because vLLM inference kernels compute logprobs slightly differently than PyTorch Hugging Face forward passes, nanoRL recomputes old_logp on the trainer under the exact sampling weights. In benchmark runs, using uncorrected vLLM logprobs caused ratio drift from 1.006 to 1.165 and collapsed accuracy from 50.0% to 18.8%.
Try it in 2 minutes
uv pip install torch gymnasium numpy transformers peft pyyaml vllm
python train.py --task cartpole --algo reinforcebash
✓ When to use
- Learning or prototyping custom RL post-training algorithms for LLMs without Ray overhead
- Running GRPO/PPO fine-tuning on consumer or multi-GPU instances using vLLM and LoRA
- Building transparent custom reward functions for mathematical or domain-specific reasoning tasks
✕ When NOT to use
- Large-scale enterprise workloads requiring Megatron-style 3D parallelism or MoE partitioning
- Multi-tenant cluster scheduling with complex UI dashboards and job managers
What to do today
- Clone alex000kim/nanoRL to inspect algos.py and tasks.py for lightweight RL implementation.
- Run python train.py --task cartpole --algo reinforce locally to verify environment setup.
- Define custom reward functions in tasks.py to post-train small LLMs on domain reasoning datasets.
Sources