ppetrichor/worlddirector

WorldDirector is a research project that generates controllable video simulations of a world, built to keep track of objects over long stretches of video even when those objects leave the frame and come back later. It comes from a group of researchers and is tied to a published paper, a project website, and a set of downloadable model weights. The approach splits the work into two stages. First, a separate large language model plans out where objects and the camera should move in 3D space over the course of the video, producing a JSON file describing every object's motion and the camera's path. Second, that plan gets converted into 2D controls and fed into a video generation model, which produces the actual footage. A technique called appearance binding helps the system remember what an object looked like, so if it moves off screen and returns, it still looks the same. Using this project requires real technical setup. You need Python, a specific version of the PyTorch library, and a package called FlashAttention, plus the WorldDirector model weights downloaded from Hugging Face. Generating video also needs multiple GPUs, since the example command in the README runs across eight of them. The full pipeline involves several manual steps: planning object motion with an LLM, preparing video segments and prompt files, and then running a generation script with many configuration options. This is not a beginner tool or a plug-and-play app. It is aimed at researchers and engineers working on video generation and world models who want to reproduce or build on the WorldDirector method. The README currently states that the full inference code and the 14-billion-parameter model have been released, with example results and demo videos included to show what the output looks like. The full README is longer than what was shown.