horizon-llm/resd

RESD, short for Reflection Enhanced Self Distillation, is a Python research project from a group called horizon-llm that implements a method for training AI language models to learn more effectively from their own mistakes. It builds on top of two existing training frameworks, called veRL and SDPO, and accompanies a research paper describing the approach. The core idea is that during training, the system keeps two forms of memory alongside the model itself. One is a playbook of reusable lessons distilled from past failed attempts, and the other is an optional buffer that stores examples of successful attempts when they happen. After each training step, the system updates this memory by removing outdated or low value entries and adding new lessons generated by reflecting on what went wrong. A teacher version of the model, kept in sync with the student model through a smoothing process, then uses this enriched context to guide the student's next round of learning, rather than only supplying a plain right or wrong signal. The project is evaluated on four benchmark tasks covering different problem types, including writing small programs to match patterns, simulating simple physics scenarios, and tagging named entities in financial documents, comparing RESD's results against the SDPO and GRPO training methods it builds on. Getting the code running requires either building a Conda environment from a provided configuration file or pulling a prebuilt Docker image, along with a Weights and Biases account and API key for tracking training runs. This is aimed at machine learning researchers already familiar with reinforcement learning style training pipelines and GPU based infrastructure rather than casual users or beginners, since running any of the included examples requires meaningful compute resources and machine learning expertise. The README does not clearly state a license for the code.