hkust-c4g/domainshuttle

DomainShuttle is an AI research project from Hong Kong University of Science and Technology that generates videos from text descriptions while keeping a specific subject (like a person, object, or stylized character) looking consistent throughout, even when placed in visually different settings or art styles. The challenge this addresses is that existing text-to-video systems struggle when you want a subject from one visual domain (say, a cartoon character) to appear in a different domain (say, a photorealistic landscape), or when a real photo subject needs to appear in an animated-style video. DomainShuttle handles this by separating how a subject looks from the domain, meaning the visual style and environment of the video, learning each independently before combining them during generation. The system is built on top of Wan2.2, a 14-billion-parameter video generation model. You give it a reference image of your subject and a text prompt describing the scene or action, and it generates a short video where that subject appears in the described setting. Setup requires a CUDA-enabled GPU, the conda environment manager, and downloading two large model files from HuggingFace. After installing dependencies with a setup script, you run a single shell script to generate videos. Sample test cases are included in the repository, and you can swap in your own reference image by editing a JSON config file. The model weights and code are licensed under Apache 2.0. A technical report describing the method in full is available on arXiv.