alex-nlp/denoiserl

DenoiseRL is the official code release for a research paper from Fudan University about a new way to train AI reasoning models. Most current approaches for improving a model's reasoning ability either depend on having access to a more powerful teacher model to imitate, or require significant effort to curate difficult training examples. DenoiseRL takes a different route: it intentionally feeds a model the first part of a wrong answer produced by a weaker, cheaper model, then trains the model to recognize the mistake and still arrive at the correct solution. The idea is similar to how noise-removal techniques work in other areas of machine learning. A weak model generates an incorrect solution to a math problem. The system takes the first portion of that wrong solution and hands it to the model being trained, telling it to continue from that corrupted starting point. The model then earns a reward if it recovers and finds the right answer. Over many such training steps, the model becomes better at catching and correcting reasoning errors mid-stream. The training process mixes two types of examples in each step: normal problems solved from scratch, and the corrupted-prefix recovery problems. Sharing a single scoring baseline across both types keeps the training stable, because recovery attempts naturally produce contrasting signals for problems the model would otherwise solve easily. Gradients only flow through the model's own continuation, not through the off-policy prefix from the weaker model, which the authors found critical for avoiding instability. Results are reported on standard math benchmarks including MATH500, AMC23, AIME24, and AIME25, using Qwen3-4B and Qwen3-8B language models as the policy being trained. DenoiseRL consistently outperforms the baseline GRPO and DAPO training methods on average across those benchmarks, with modest extra training time per step. The code is built on a local fork of the verl reinforcement learning framework. Setup requires creating a Python virtual environment, installing pinned dependencies, and registering the local framework in editable mode. Training scripts are provided for 1.7B, 4B, and 8B model sizes.