existentialrobotics/serf-mapping

SERF-mapping is the official code release for a research paper titled "SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation." It provides the mapping component of a system designed to help mobile robots perform long sequences of manipulation tasks, such as picking up and placing objects across a large environment over extended periods. The core idea is to build two types of maps from camera footage: one capturing the surrounding environment (furniture, objects, spatial layout) and one capturing the robot itself. These are called "neural point" maps, meaning they store learned visual features at 3D positions rather than raw color or depth values. The features come from a vision model called DINOv3, which can connect visual appearance to language, allowing the robot to find objects by their name or description. By updating these maps with expert demonstrations, a robot can learn where things are and how to find them again during a long task. The repository covers three workflows: building a SERF map from a dataset of recorded episodes, updating an existing map using a replayed expert demonstration with a tracking algorithm called CoTracker, and visualizing the resulting feature map with or without a 3D robot overlay. Datasets and pre-trained models are hosted on Hugging Face and can be downloaded with a few commands. The code also includes a Python script to extract DINOv3 visual embeddings from HDF5 data files before training. This is academic robotics research code. It depends on conda or mamba for environment setup, requires downloading large model weights and datasets from Hugging Face, and is part of a larger system. The companion repository SERF-VLA contains the planning and execution component that uses these maps at runtime. The license is specified in the LICENSE file in the repository.