emotion-machine-org/personaplex-finetune

This is an unofficial pipeline for fine tuning PersonaPlex, a variant of the Moshi 7B voice model with voice and role conditioning, on specific professional domains such as pharmaceutical patient support or insurance brokerage calls. The pipeline works in several stages. First, it uses Claude to generate synthetic dialogue scripts between an agent and a client, including structured context injection markers. These scripts are converted to audio using VibeVoice, a text to speech system, producing stereo recordings with the agent and client on separate audio channels. The audio is aligned using WhisperX, a speech recognition and alignment tool, and the dataset is then used to fine tune PersonaPlex using LoRA, a technique that trains a small set of additional weights rather than retraining the full model, with FSDP across multiple GPUs. At inference time, an external puppeteer language model injects context tokens into the model's text stream through a hidden channel, allowing the model to receive live grounding information mid conversation without that context appearing in the output. The authors ran 84 training runs on insurance and pharmaceutical datasets totaling roughly 3,800 and 2,000 synthetic dialogues respectively. They document five recurring failure modes and the two fixes that mattered most: setting text_padding_weight to 0.0 to eliminate a PAD token collapse failure, and applying L2 regularization on LoRA B weights to prevent catastrophic forgetting. Requirements include Python 3.10 or newer, 3 to 5 GPUs with 24 GB VRAM each, RTX 4090 tested, or a single H100, plus an Anthropic API key for dialogue generation. The project is described by its authors as a work in progress, not yet tested on a wide range of setups. The full README is longer than what was shown.