yanqinjiang/omnix

OmniX is the official code release for an academic research paper about rebuilding moving, three dimensional scenes from video, a task researchers call 4D reconstruction because it captures both space and time. The project comes from researchers at CASIA, Tencent Hunyuan, and HKUST, and it was accepted at a major computer vision conference. The core idea is to take footage from any set of camera views and any point in time, and predict how the scene looks and moves in 3D, using a single trained neural network rather than needing to run a separate optimization process for each new scene. The team frames this as learning trajectory fields, a way of describing how points in the scene move over time, and their model produces this in a single forward pass rather than a slow iterative one. This is research code, not a consumer product. Using it requires a capable Nvidia GPU, specifically ones from Nvidia's newer Hopper or Blackwell generations, since the code depends on a specialized attention technique called FlashAttention-3 that needs that hardware. Setup involves creating a Python environment, installing a matching version of PyTorch for your CUDA version, and compiling two additional pieces of code from source, including a specialized attention library and detection operators. Once set up, you can either run inference using a pretrained model checkpoint the authors provide to reconstruct a sample scene from images, or train the model yourself on your own data after preprocessing it with tools borrowed from a related project. The authors note that the provided training settings are a starting point and will likely need adjustment for different hardware or datasets. The project builds on and credits several other open source research efforts, and the authors ask that anyone who finds it useful cite their paper.