anliyuan/feathertalk

FeatherTalk is a research project for building lightweight talking head videos, where a still face is animated to speak in sync with an audio clip. It is a cleaned up and extended version of an earlier project called Ultralight Digital Human, and it focuses on training a personalized model from a short video, using smaller audio processing so it can eventually run on mobile devices. The main new piece is a component called FeatherHuBERT. Digital human systems like this normally use a large speech recognition model called HuBERT to turn audio into a feature representation that drives mouth movement, but that model is heavy and more powerful than the task actually needs, since no text recognition is required, only a stable signal of what the mouth should be doing. FeatherHuBERT is a much smaller version trained to copy HuBERT's output while using roughly forty times less compute, and it keeps the same input and output shape so the rest of the training and inference pipeline does not need to change. Other additions include support for a second, mobile-friendly network design called MobileOne alongside the original one, extra training losses that specifically target the mouth region to reduce jitter around the lips, removal of a component called SyncNet from the default training path, and support for resuming training from a checkpoint and exporting models to the ONNX format for faster deployment. To try it, the project provides a downloadable demo package containing a short training video and a pretrained FeatherHuBERT checkpoint, which can be preprocessed with a provided Python script to quickly test the training flow before using your own video. This project is aimed at researchers and developers working on talking head or digital human generation who want a smaller, more deployment-friendly alternative to the heavier original pipeline.