aliaksandrsiarohin/first-order-model

This repository contains the research code for an academic paper called First Order Motion Model for Image Animation, published at NeurIPS by Aliaksandr Siarohin and four co-authors. The point of the project is to take one still image of a subject and a separate video of something or someone moving, then produce a new video in which the subject from the still image moves the way the subject in the driving video does. The README shows example results for three datasets: VoxCeleb of talking faces, a Fashion dataset of people modeling clothes, and an MGIF dataset. The technique works without knowing in advance what the subject is. You do not have to label parts of a face or a body. The model learns its own set of keypoints during training and then transfers the motion of those keypoints from the driving video onto the source image. The README describes two ways to run the animation step: an absolute mode that copies the driving keypoints directly, and a relative mode that uses only the change in keypoint positions and applies that change to the source. The relative mode usually looks better but needs the first frame of the driving video to have a similar pose to the source image. The project is written in Python 3 with PyTorch, with one YAML configuration file per dataset. The README explains how to install dependencies, how to download pre-trained model files from Google Drive or Yandex Disk, and how to run a single command-line demo that produces a result.mp4 file. A helper script suggests crop boxes for a YouTube video using ffmpeg, and there is also a Docker image with nvidia-docker support if local library versions cause trouble. There are notebooks for running the demo on Google Colab and Kaggle, including a graphical interface contributed by a community user. The README also covers training your own model on one of the supported datasets such as Bair, MGIF, Fashion, and Taichi, evaluating reconstruction quality, and points to a follow-up project that adds support for face-swap and articulated objects.