hpcaitech/open-sora

Open-Sora is an open-source Python project for generating videos from text descriptions using artificial intelligence. You type a prompt describing a scene, and the model produces a short video clip. The goal of the project is to make high-quality AI video generation accessible and affordable, the team reports training their 11-billion-parameter model for roughly $200,000, significantly lower than what comparable closed systems cost to develop. The project covers the full pipeline for working with video AI: preprocessing video data for training, training the model itself with efficiency optimizations, and running inference to generate new videos. It supports generating videos of varying lengths and resolutions, from short clips at lower resolutions to 5-second clips at 1024×576 pixels. The system also supports image-to-video (animating a still image), video-to-video (transforming an existing clip), and text-to-image generation. Under the hood, Open-Sora 2.0 uses an 11-billion-parameter model architecture and incorporates a component called a 3D-VAE (a type of encoder that compresses video data for efficient processing), along with rectified flow training (a technique for improving the quality of generated outputs). The model weights and training code are published openly. A researcher studying AI video generation, a developer building a video creation product, or a team wanting to fine-tune a video model on their own dataset would turn to this repository. It runs on Python and requires GPU hardware for training and inference.