yangtiming/fast-sam-3d-body

Fast SAM 3D Body is a research project from a team at USC, UC San Diego, NVIDIA, and Meta Reality Labs that makes an existing computer vision model, called SAM 3D Body, run much faster. SAM 3D Body can look at a single photo of a person and reconstruct a 3D model of their whole body pose, but the original version takes several seconds per image, which is too slow for anything that needs to react in real time. Fast SAM 3D Body reworks how the model processes an image so it can produce the same kind of 3D body reconstruction many times faster, without retraining the underlying model. The project reports up to a 10.9 times overall speedup compared to the original SAM 3D Body, and a much larger speedup, over 10,000 times, on one specific step that converts the raw mesh output into a simpler joint based format used by robotics and animation tools. On a high end NVIDIA RTX 5090 graphics card, the full pipeline runs in about 65 milliseconds per frame, fast enough to track a person's body movement live from a single camera. The authors demonstrate this speed by using Fast SAM 3D Body to control a Unitree G1 humanoid robot in real time, letting the robot copy a person's body movements from ordinary video, including gestures, walking, squatting, and kneeling. They also show it can be used to collect example movements for training robot behavior policies, reporting an 80 percent success rate on a box grasping task using just 40 collected examples. This is an advanced research codebase intended for people already working in computer vision or robotics. Setup involves a provided environment script, downloading model checkpoints, and optionally converting models with TensorRT for extra speed. There is an accompanying academic paper describing the technical details of the acceleration methods used.