lllyasviel/toondecompose

ToonDecompose takes a cartoon animation and automatically breaks it apart into separate moving pieces, called "sprites." You give it a sequence of frames from an animation and tell it how many objects or components you want. It gives you back each component as its own sequence of images, along with data describing how each one moves over time. The whole process is automatic, you don't need to manually outline or mask anything. The tool works in stages. First it figures out how every pixel moves between frames using a technique called optical flow. Then it analyzes those movement patterns to guess where the boundaries of each object are. After that, it estimates how each sprite shifts, rotates, or scales from frame to frame. Finally, it refines all of these estimates together to produce clean, separated sprite layers. The whole pipeline involves running several Python scripts in sequence, and processing a 78-frame clip takes roughly 40 minutes on a decent GPU. This would be useful for animators, game developers, or researchers who want to pull individual elements out of an existing animation, for example, separating a running character from the background so each layer can be edited, reused, or repositioned independently. The project includes a real test case using 78 frames from the anime "Violet Evergarden," chosen because it's a genuinely challenging clip with a character in fast motion. One notable aspect is that the results are sensitive to your GPU model. The author tested several cards and found that some produce identical results while others deviate, and the cause isn't fully understood. You need a CUDA-enabled GPU, CPU-only mode doesn't work. The methods behind the tool are described in a SIGGRAPH Asia 2022 paper by the same author, and the project is still being actively refined.