deftruth/audio2face

Audio2Face is a project that takes an audio recording of someone speaking and automatically generates realistic facial animation for a digital character. The main benefit is that you can make a 3D avatar talk and express emotions without manually animating every mouth movement, saving significant time and effort in character animation. At a high level, the system works by analyzing the sound of a voice and translating it into "blendshape weights," which are essentially the numerical controls for different facial muscle movements. The process involves breaking down the audio into distinctive sound features, analyzing how those sounds relate to mouth and face movements, and applying an emotional state to the mix. The result is a set of instructions that tells a 3D character exactly how to move its face to match the spoken words. This tool would be useful for game developers, animators, or anyone building virtual assistants and digital humans. For example, if you are creating a virtual customer service agent or a non-player character in a game, you could just provide a voice recording and the character's face would automatically move in sync with the audio. The project includes a test application where you can speak into a microphone and watch a digital character named Xiaomei respond with matching facial animations in an Unreal Engine project. To use the system, you need some specific setup. Training the model requires recording video of a person speaking, creating matching facial animation in Autodesk Maya, and processing the audio into a format the system can learn from. The creators recommend using their separate Avatary software to generate training data quickly. They also provide a pre-trained model so you can test the talking character without building one from scratch. The project relies on an older version of TensorFlow (1.15) and is based on research originally published by NVIDIA. It is released under the MIT license, meaning it is free to use and modify.