localai-org/talking-portrait.cpp

``json { "i": 0, "repo": "localai-org/talking-portrait.cpp", "stars": 4, "lang": "C++", "ok": true, "char_count": 1610, "text": "This project animates a still portrait photo using a voice recording. You give it one 512x512 portrait image and a 16 kHz speech audio file, and it produces a 25 frame per second talking head video. It is a C++ port of an existing research project called FLOAT, built to run on CPU or NVIDIA GPUs using the GGML tensor library.\n\nThe software pipeline works in stages. First, an audio encoder processes the speech signal. Then a motion generator uses a technique called flow matching to predict how the face should move frame by frame. Finally, a motion decoder warps and renders the portrait image according to those predictions, producing the animated video. The caller specifies the emotion as a simple label, and the system does not attempt to detect emotion from the audio itself.\n\nThe repository provides a command line tool that takes a portrait image, a WAV file, and an emotion label, and outputs a video file. It uses ffmpeg for video encoding. Build requirements include CMake, Ninja, and a C++17 compiler, with optional Vulkan support if the relevant headers and tools are installed. There are also presets for debugging, fuzzing, and CUDA builds.\n\nBeyond the core tool, the project includes a WebGL gallery demo. This demo lets users upload a portrait and either an audio file or a microphone recording, then plays the generated talking portrait video inside a virtual 3D gallery rendered in the browser. Completed portraits hang along a virtual wall, and the camera tours them in order. The demo runs locally through a Docker container with a Go server and HTTPS termination via Caddy. It has no authentication, so the documentation advises restricting network access.\n\nThe project tracks its progress through a series of milestones. It reports passing several parity gates against the original PyTorch implementation, with some limitations in half precision mode. It also claims performance above its targets and zero memory allocations per frame on the generation thread. The full README is longer than what was shown.", "error": null } ``