huggingface/alignment-handbook

The Alignment Handbook is a collection of training recipes published by Hugging Face for turning a base language model into a helpful, safe assistant. A base language model is trained to predict text but does not know how to follow instructions or have a conversation. Alignment is the process of further training that model to behave the way users and developers want, for example by following instructions, avoiding harmful responses, or adopting a particular tone. The repository provides scripts and configuration files that cover the main stages of this process. The first stage is supervised fine-tuning, where the model learns to follow instructions by training on examples of good responses. The second stage is preference alignment, where the model learns to prefer better responses over worse ones using techniques called DPO (Direct Preference Optimisation) and ORPO. The repository also includes scripts for continued pretraining, which is useful for adapting a model to a different language or a specialized domain, and for reward modeling. Each recipe is a YAML configuration file that captures all the settings for a single training run. The repository ships recipes for several publicly known models, including the Zephyr series and SmolLM. These recipes let researchers reproduce those models or adapt the configurations for their own training runs. The scripts support distributed training across multiple GPUs using a library called DeepSpeed, as well as lighter-weight parameter-efficient fine-tuning approaches called LoRA and QLoRA that work on smaller hardware. Installation requires Python 3.11, a specific version of PyTorch matched to the CUDA version on your machine, and Flash Attention 2. The project is developed by the Hugging Face H4 team and is intended for researchers and engineers who want to train their own aligned language models rather than just use existing ones. The full README is longer than what was shown.