h6kplus/phymotion

PhyMotion is a research project that tackles a specific problem in AI video generation: when AI systems generate videos of people moving, the results often show physically impossible motion, such as bodies that float, limbs that overlap each other, or movements no real person could perform. The scoring signals normally used to judge video quality rely on 2D visual impressions and often rate these unrealistic videos highly anyway. PhyMotion introduces a structured reward signal that judges human motion along three axes: kinematic plausibility, meaning how joints move and accelerate, contact and balance consistency, meaning whether feet correctly touch the ground, avoid sliding, and keep balance, and dynamic feasibility, meaning whether the forces and torques involved are physically possible. To compute this reward, the system recovers three-dimensional body shape estimates, called SMPL meshes, from generated video frames, retargets that motion onto a humanoid character inside the MuJoCo physics simulator, and scores the result across all three axes. This reward is used during reinforcement learning post-training, a process where a video generation model that has already been trained gets further refined by rewarding the outputs that score well on this physical realism check. Compared with existing reward methods, training with PhyMotion produced more consistent improvements in motion realism, including a gain of 68 Elo points as judged by blind human evaluators. The project is written in Python and is meant to plug into existing video generation training pipelines built on models like Wan2.1. Setting it up involves installing PyTorch with CUDA support, MuJoCo, and SMPL-X body models, plus downloading multi-gigabyte pretrained checkpoints from Hugging Face for the base video model and the reward's pose recovery component. Pretrained LoRA weights and a dataset of motion prompts built from MotionX are provided for people who want to reproduce or build on the results without training from scratch. The full README is longer than what was shown.