yuecheng919/gemdepth

GemDepth is a research framework for estimating depth from video. Depth estimation means figuring out how far away objects in a scene are, based purely on camera footage, turning a flat 2D video into something that understands the 3D structure of the world. The particular challenge it tackles is temporal consistency: most depth estimation methods process each video frame independently, so the depth values flicker or jump between frames even when the scene itself is stable. GemDepth solves this by making the model aware of how the camera is moving between frames. It does this through two main innovations described in the paper. First, a Geometry-Embedding Module predicts the camera's motion between frames and encodes that information into the model's processing, giving it an explicit understanding of 3D geometry rather than making it guess blindly. Second, an Alternating Spatio-Temporal Transformer uses that geometric awareness to find correspondences between points across frames, improving both the sharpness of depth edges and the smoothness of depth values over time. The model claims state-of-the-art results on standard benchmarks including Sintel, KITTI, Bonn, and ScanNet, particularly in scenes with moving objects. You would use this for applications that need reliable depth from video: 3D scene reconstruction, robotics, autonomous driving research, or visual effects. It is written in Python using PyTorch and requires significant GPU memory (15 to 44 GB depending on settings). Pre-trained weights are available on HuggingFace, and a Gradio web interface is included for testing without writing code. The work was accepted at ICML 2026.