zhengdian1/interleavethinker

InterleaveThinker is a research project that teaches AI image generators to produce sequences that mix text and pictures in a coordinated way, rather than generating one image in isolation. The problem it addresses is that most image generation models take a single prompt and produce a single image. This project adds the ability to plan out a longer sequence where text and images alternate, like the pages of a visual story or a step-by-step illustrated guide. The system works by adding two specialized AI components on top of any existing image generator. A planner agent decides how to structure the text and image inputs for each step in the sequence. A critic agent then reviews the generator's output, spots where it went wrong, and rewrites the instructions to correct course. Together they form a loop that can handle multi-step tasks where each image needs to be consistent with what came before. The project was published as an academic paper in June 2026. It includes three training datasets built for this task, covering about 200,000 examples in total, as well as trained model checkpoints for the planner and critic that are available on Hugging Face. The training process uses a technique called reinforcement learning with a reward function that evaluates both accuracy and whether the step-by-step corrections actually improve results. Benchmarks in the paper show significant gains on standardized tests for this kind of interleaved generation. Setting up the system requires separate Python environments for inference, supervised fine-tuning, and reinforcement learning training. Image generation is handled by a local API service, with support for models like FLUX and Qwen Image. Multi-machine deployment with load balancing is also documented for higher-throughput use. The intended applications include visual storytelling, illustrated how-to guides, and annotating long sequences of actions for robotics or similar tasks.