carpedkm/customdit

This repository is a research project, published at ICASSP 2026, that focuses on generating videos of specific people or objects based on a reference image and a text description. The core problem being solved is called "customized video generation": given a photo of a specific subject (a person, a pet, a product), produce a new video showing that same subject doing something described in text. The challenge is that existing systems are limited by the lack of large, publicly available training data for this task. The project introduces three related contributions. The first is PexelsCustom-1M, a dataset of one million video clips sourced from the Pexels platform, paired with reference images of the subjects appearing in each clip and text descriptions of what is happening. The dataset spans over 8,000 categories. The actual video files are not included in the repository since they are hosted by Pexels, but the metadata, reference images, and annotations will be released on HuggingFace. The second contribution is CustomDiT, the video generation model itself. It is built on top of an existing text-to-video model called CogVideoX and adds the ability to condition generation on a reference image by injecting information about the subject's appearance into the model's attention layers. Training runs in two stages: the first stage teaches the model to follow the reference image, and the second stage adds data augmentation to prevent the model from simply copying the reference image directly into the output. The third contribution is OpenCustom, a benchmark for evaluating how well these systems work. It covers more than 1,000 categories, which is significantly broader than the 100-category benchmarks previously available. The evaluation measures both how closely the generated video resembles the reference subject and how well it matches the text description, plus motion quality metrics. The repository includes code for all three components: model training and inference, the data curation pipeline that processed raw Pexels videos into training data, and the evaluation framework. Three separate Python environments are required for the different components.