lllyasviel/controlnet

ControlNet solves a real creative problem: when you use AI image generators like Stable Diffusion, you can describe what you want in text, but you have very little control over the exact composition, pose, or structure of the result. ControlNet adds a way to guide image generation using visual signals, things like edge outlines, human body poses, depth maps, or hand-drawn scribbles, so the AI generates images that follow your provided structure, not just your words. The way it works is clever: it makes a copy of part of the image-generation neural network. One copy is "locked" and stays unchanged (preserving the original model's capability), while the other copy is "trainable" and learns to respond to your extra visual condition. These two copies are connected through special "zero convolution" layers, small 1x1 filters initialized to output nothing at the start, which means the system begins training without causing any disruption to the original model. As training continues, these connectors gradually learn to inject the visual condition into the generation process. You would use ControlNet when you want to generate an image that matches a specific pose, follows the edges of a sketch you drew, mirrors the depth structure of a reference photo, or replicates the layout from a line drawing. Instead of prompting and hoping, you get reproducible control. The stack is Python, built on top of Stable Diffusion 1.5 (the popular open-source image model), and uses Gradio to provide interactive browser-based demos. Supporting tools include OpenPose for body detection, Midas for depth, and various edge-detection algorithms. Training can run on consumer GPUs with limited memory.