lukashoel/video_to_world

video_to_world is the official research code for a paper called World Reconstruction From Inconsistent Views, which tackles a specific problem in AI generated video. When a video diffusion model, an AI system that creates video clips from a prompt, generates a sequence showing a moving camera through a scene, the frames it produces are not perfectly consistent with each other in 3D. Small errors accumulate, so if you try to reconstruct a real 3D model of the scene directly from those frames, the result looks warped or broken. This project introduces a method that corrects for those inconsistencies using what it calls non rigid alignment, bending and adjusting the frames so they line up into one coherent 3D world instead of assuming they were already perfectly consistent. The pipeline works in clearly defined stages. First, it estimates a rough 3D point cloud for each frame of an input video using a separate model called DepthAnything 3. Next, it iteratively aligns all of those per frame point clouds into one shared canonical scene, correcting for the small shifts and distortions between frames. A further global optimization step refines this alignment across all frames at once, and later stages train the scene as a Gaussian Splatting representation, a technique for rendering realistic 3D scenes efficiently, which can be viewed from new camera angles. Running this project requires a capable NVIDIA GPU, a conda based Python environment, and installing several other research codebases as dependencies, including DepthAnything 3, gsplat, tiny cuda nn, and RoMaV2, some of which need small patches applied before they work correctly together. There are two built in presets, a faster mode and a more extensive mode with more optimization steps and higher quality output. This is a research tool for people working in 3D reconstruction or computer vision, not a general purpose app. It accompanies an academic paper with a project page and demonstration video linked from the README for readers who want the full technical background.