kidding-404/nano-verl

nano-verl is a training framework for reinforcement learning with language models. It is designed to mirror a larger system called verl, but at a much smaller scale: nano-verl has around 6,000 lines of code versus the 90,000-plus lines found in the original. This makes the codebase easier to read and understand if you want to study how this kind of training works under the hood. The framework handles distributed training, which means it can spread computation across multiple machines or GPUs simultaneously. It uses two underlying systems to do this: one called FSDP for training and one called vLLM for generating text, with a tool called Ray coordinating work across machines. It also supports asynchronous training, where the model can continue learning from slightly older data rather than waiting for every step to finish before moving on. To install it, you clone the repository, use a package manager called uv to install dependencies, and then separately install a performance library called flash-attn. The README includes configuration files for running quick experiments on publicly available math datasets using small language models. The project includes benchmark results showing that a model trained with nano-verl reaches accuracy scores on math reasoning tests that are comparable to the same model trained with the full verl framework. This suggests the smaller codebase does not meaningfully sacrifice training quality. This project is aimed at researchers or developers who want a simpler codebase to experiment with or learn from, rather than dealing with the full complexity of production-scale reinforcement learning systems. If you already know what distributed GPU training is, this is a stripped-down reference implementation. If you do not, the README is technical and assumes familiarity with these tools.