mimic-video/mimic-video

mimic-video is a research project that teaches robots to perform tasks by learning from video, instead of relying only on hand labeled robot demonstrations. The core idea is to take a video prediction model, a system already trained to understand how the physical world moves and changes over time, and attach a small extra component that turns that understanding into robot actions. This extra component is called an action decoder, and it can be trained without retraining the large video model itself, which saves a large amount of computing power. The project calls the resulting systems Video-Action Models, or VAMs for short. The authors describe them as a step beyond a more common approach called VLAs, which pair vision and language models directly with robot control. By building on a video model instead, mimic-video aims to give robots a better sense of real world cause and effect, since video models already learn how objects move, fall, and interact just from watching footage. The repository ships trained checkpoints for two established robotics benchmarks, called Bridge and LIBERO, built on top of a lightweight video model named Cosmos Predict2. It includes code for three main stages of the pipeline: preparing training data, training and running the model, and evaluating results in simulation. Setup uses the uv Python package manager, and training relies on torchrun for running across multiple GPUs or multiple machines. This is a research codebase aimed at people already working in robot learning or machine learning research, not a beginner friendly tool. Getting it running involves downloading large model checkpoints, setting up simulation environments, and following multi step data preparation scripts for each benchmark. There is an accompanying paper and project website linked from the README for readers who want the full technical details behind the approach.