cambrian-mllm/cambrian-p

Cambrian-P is a research AI model from New York University, UC Berkeley, and Meta FAIR that can watch a video and answer questions about the spatial layout of the scene while simultaneously figuring out how the camera was moving during the recording. Most AI models that work with video treat it as a sequence of images and focus on what objects appear. Cambrian-P goes further by also tracking the camera's position, rotation, and field of view for each frame, giving it a richer understanding of depth and 3D structure. The model is built on top of an earlier architecture called Cambrian-S, which combines a vision-processing component (SigLIP2) with a language model (Qwen2.5, 7 billion parameters). Cambrian-P adds a small extra piece: one learnable camera token per video frame that teaches the model to reason about physical camera motion. A separate lightweight module then converts that learned information into concrete numbers describing where the camera was pointed and how far it moved between frames. In practice this means the model can do two things at once in a single pass: answer natural-language questions about a scene (like which direction an object is relative to another, or how many items are on a shelf), and output a trajectory describing how a recording device moved through the space. On a spatial video reasoning benchmark called VSI-Bench, the 7B version scores 73.7 percent average accuracy, which the authors report is the best result among models of similar size. On camera tracking benchmarks using real indoor recordings from ScanNet, it matches or beats streaming methods that use more specialized components. The repository contains training code, evaluation scripts, five pre-trained model variants available on Hugging Face, the annotated pose dataset used for training, and documentation for setting up the environment and preparing data. Training requires significant compute: it fine-tunes from an existing checkpoint using roughly 236 GB of video data plus additional pose annotation files. The code is written in Python and uses PyTorch with CUDA support.