allenai/open-instruct

Open Instruct is a research codebase from the Allen Institute for AI that focuses on teaching AI language models to follow instructions. Language models start out knowing a lot about text from their initial training, but they need an additional step to learn how to respond helpfully to requests in conversation. That additional step is called post-training or instruction-tuning, and this repository collects the code and methods for doing it using publicly available data and models. The project covers three main training approaches. The first is supervised fine-tuning, where the model learns from examples of good question-and-answer pairs. The second is preference training, where the model is shown pairs of responses and learns which one is better based on human or automated feedback. Two specific methods used for preference training are called DPO and PPO, and the project has published research papers comparing how each one works in practice. The third approach is reinforcement learning with verifiable rewards, which trains the model to optimize for outputs that can be checked for correctness, such as math answers. The team has used this codebase to train and release a family of models called Tulu, including versions built on top of Llama 3.1 and OLMo 2, another open AI model from the same institute. Those trained models are freely available to download. The README links to a free demo where anyone can try one of the resulting models without setting anything up. From a practical standpoint, the project is aimed at AI researchers and engineers who want to replicate, study, or build on these training techniques. The README provides setup instructions using a Python package manager and notes that the codebase is a research project, meaning it does not promise backward compatibility across versions. The project is backed by several academic papers that describe the experiments behind each training approach. The most recent release, called TULU 3, covers the full post-training process for both Llama 3.1 and OLMo 2 models.