bytedtsinghua-sia/direct-opd

Direct-OPD is a research code repository from a team at Tsinghua University and ByteDance, focused on a machine learning training technique called Direct On-Policy Distillation. The idea addresses a specific problem in reinforcement learning: how to take what a small teacher model learned through RL training and pass that improvement on to a bigger, stronger student model, without simply making the student copy the teacher's final answers. Instead of copying, the method looks at how the teacher's behavior changed before and after its own RL training, and uses that change as a reward signal for training the student. The student generates its own responses, staying on policy, while the teacher's shift in probability is used to score how good those responses are. According to the README, this approach lets a weaker teacher model improve a stronger student by transferring the direction of improvement rather than the teacher's final capability ceiling. This is a research code release rather than a polished tool. It contains the training scripts used for one specific experiment, moving learned improvements from a model called JustRL onto a Qwen model, built on top of a modified version of an existing training framework called verl. Setup requires a Python environment through conda, installing several machine learning libraries, and downloading or symlinking model weights and training data files. Detailed setup steps live in a separate docs file referenced from the README. The repository is organized into folders for evaluation data, launch scripts, the patched training framework, and setup documentation. There is an accompanying academic paper on arXiv and trained models available on Hugging Face for those who want to use the results without running the training themselves. This project is best suited for machine learning researchers and engineers already familiar with reinforcement learning and large language model training, not casual users. It is not something a non-technical reader could install and try quickly, since getting it running needs a GPU setup, specific dependencies, and comfort reading research-oriented documentation.