gair-nlp/livetalk

LiveTalk is a research project that generates realistic talking avatar videos in real time. The problem it solves: creating a video of a person speaking, with their lips synchronized to audio, normally takes many seconds or even minutes of processing. LiveTalk brings that down to near-instant generation, fast enough to support live, back-and-forth conversation with a digital avatar. The core idea is a technique called on-policy distillation, which takes a large, slow video generation model and compresses its behavior into a much faster version that produces results in just four steps instead of dozens. The system takes three inputs: a reference photo of a person, an audio clip of speech, and a text description of how the video should look. It then generates video where the person in the photo speaks in sync with the audio, producing 24 frames per second with less than a third of a second before the first frame appears. The system is designed for conversational AI applications, for example, giving a chatbot a lifelike face that responds to you in real time. It is written in Python and requires a high-end GPU with at least 24 gigabytes of memory to run.