msalab-pku/loomvideo

LoomVideo is a research project from Peking University and Alibaba Group that generates and edits video using artificial intelligence. It is the official code release accompanying an academic paper, and it lets a single AI model take in different combinations of text prompts, existing videos, and reference images, then produce a new video as output. The project supports four related tasks through one model: creating a video purely from a text description, editing an existing video according to written instructions, editing a video using both written instructions and a reference image to guide the change, and combining several separate reference images into one coherent video following a text prompt. The README shows example outputs for each of these, such as changing the artistic style of a video while keeping its motion and camera movement intact, replacing an object in a scene with something else, or combining photos of a person and a location into a new video of that person walking through that location. Compared to other AI models that try to do all of this in one system, the authors highlight that theirs is much smaller, at 5 billion parameters versus the 13 billion or more used by comparable models, while still achieving similar or better results and running over five times faster. They describe three technical design choices that make this possible, involving how information from a language and vision model gets fed into the video generation model, and how the model is told which parts of its input are the unchanged source content versus the new content to generate. To use it, you need to clone the repository and set up a Python environment, for which the authors recommend a tool called uv for fast, reproducible installs, plus a library called Flash Attention for efficient model computation. The trained model weights are hosted separately on Hugging Face and downloaded during setup. Running the full pipeline requires a capable GPU, since this is a large video generation model, though the README's setup section does not specify exact hardware requirements.