modelscope/diffsynth-studio

★ 12,401PythonAudience · researcherComplexity · 4/5Setup · hard

Mindmap

mindmap
  root((DiffSynth))
    What it does
      Image generation
      Video generation
      Audio and music
    Supported Models
      Stable Diffusion
      MOVA video
      ACE-Step audio
      JoyAI editing
    Setup
      Python
      GPU required
      pip install
    Audience
      AI researchers
      ML developers

mindmap root((DiffSynth)) What it does Image generation Video generation Audio and music Supported Models Stable Diffusion MOVA video ACE-Step audio JoyAI editing Setup Python GPU required pip install Audience AI researchers ML developers

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Generate short videos from text prompts at 720p using the MOVA model

USE CASE 2

Create music from text descriptions using the ACE-Step audio generation model

USE CASE 3

Edit an existing image by typing a natural-language instruction using JoyAI-Image

USE CASE 4

Train a controllable image generation model using the Diffusion Templates plugin framework

Tech stack

PythonPyTorch

Getting it running

Difficulty · hard Time to first run · 1h+

Requires GPU hardware with sufficient VRAM, project is experimental so APIs may change and issue response times can be slow.

In plain English

DiffSynth-Studio is a Python library and engine for working with diffusion models, which are a type of AI system capable of generating images, videos, audio, and music from text prompts or other inputs. The project is maintained by the ModelScope Community and positions itself as an experimental playground oriented toward researchers and developers who want to explore what these generative models can do. The codebase splits into two separate projects. DiffSynth-Studio is the experimental branch, where new model types and techniques get added quickly, sometimes at the cost of stability. DiffSynth-Engine is the companion project aimed at production deployment, offering more consistent behavior and higher performance. If you want to experiment with the newest AI generation capabilities, Studio is the entry point, if you want to ship something reliable, Engine is the intended path. The range of supported models is broad. Recent additions include text-to-music generation via ACE-Step, video generation at 360p and 720p via MOVA, instruction-guided image editing via JoyAI-Image, and audio-video generation via LTX-2. Earlier models like Stable Diffusion 1.5 and SDXL are also supported for academic purposes. The project also introduced a Diffusion Templates framework in early 2026, described as a plugin system for training controllable generative models with lower setup overhead. The team is small, mainly two contributors, which the README explicitly acknowledges. New features come in regularly but issue response times can be slow. For anyone using this as a dependency in a real project, that is worth knowing up front. Documentation exists in both English and Chinese, and there is a Discord community for questions. Getting started requires Python and GPU hardware. The package installs via pip. Example scripts and per-model documentation are organized in the repo under the examples and docs directories. The full README is longer than what was shown.

Copy-paste prompts

Prompt 1

Install DiffSynth-Studio via pip and generate a 720p video from a text prompt using the MOVA model, show me a working example

Prompt 2

Use DiffSynth-Studio's ACE-Step model to generate a short music clip from a text description of a genre and mood

Prompt 3

Run instruction-guided image editing with JoyAI-Image in DiffSynth-Studio, give me a working Python script

Prompt 4

How do I use the Diffusion Templates framework in DiffSynth-Studio to set up a training run for a new controllable generative model?

Prompt 5

What is the difference between DiffSynth-Studio and DiffSynth-Engine and which should I use if I want to ship this in a real product?

Open on GitHub → Explain another repo

← modelscope on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.