g3t-paper/g3t

This is a research codebase from Cornell University presenting a system called G3T that reconstructs three-dimensional scenes from photographs. Existing 3D reconstruction tools often produce results that are tilted or inconsistently oriented because photos can be taken from any angle. G3T addresses this by producing results that are always aligned to gravity, meaning the reconstructed scene is automatically oriented upright regardless of how the input images were captured. The core model is a transformer, a type of neural network, that takes a collection of photos and predicts a pointmap for each one. A pointmap assigns a three-dimensional position to every pixel in an image, effectively describing the shape of the scene in space. Because G3T applies a gravity-aligned coordinate system before making its predictions, the output structure is consistently upright without any separate correction step afterward. The repository also includes G3T-Long, an extension for handling long video sequences. Instead of treating a whole video as one reconstruction problem, G3T-Long breaks the sequence into overlapping chunks, reconstructs each with G3T, and then joins them into a single model. A loop closure mechanism can detect when the camera returns to a previously mapped location and uses that to correct drift that accumulates over long sequences. Both tools download pre-trained model weights from Hugging Face automatically on first run. The standard workflow is to point a script at a folder of images, run inference, and then explore the result in a browser-based interactive visualizer. The project builds on earlier open-source 3D reconstruction work and was fine-tuned from an existing checkpoint. Training code and a faster C++ loop closure solver are planned but not yet released.