orange-3dv-team/smartdirector

★ 13Audience · researcherComplexity · 5/5Setup · hard

Mindmap

mindmap
  root((SmartDirector))
    What it does
      Keyframe-guided video gen
      Cinematic clip production
      Narrative timing control
    Pipeline Stages
      Director-Gen low-res pass
      Director-SR sharpening pass
    Video Modes
      Single continuous shot
      Multi-shot sequence
      Video extension
    Training Data
      Movie scene extraction
      Single and multi-shot clips
    Status
      Code pending compliance review
      Paper on arXiv 2605.27891
      CAS and HUST teams

mindmap root((SmartDirector)) What it does Keyframe-guided video gen Cinematic clip production Narrative timing control Pipeline Stages Director-Gen low-res pass Director-SR sharpening pass Video Modes Single continuous shot Multi-shot sequence Video extension Training Data Movie scene extraction Single and multi-shot clips Status Code pending compliance review Paper on arXiv 2605.27891 CAS and HUST teams

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Generate a multi-shot video sequence where each scene transition is anchored to a specific keyframe you provide at a chosen time position.

USE CASE 2

Extend an existing video clip using SmartDirector's keyframe-conditioned generation to continue the narrative beyond the original ending.

USE CASE 3

Apply the Director-SR stage to sharpen a rough low-resolution video draft using the original high-resolution keyframes as detail-recovery anchors.

USE CASE 4

Reproduce the SmartDirector evaluation from the paper on a custom movie sequence using the provided data extraction pipeline.

Getting it running

Difficulty · hard Time to first run · 1day+

Code and dataset are pending a corporate compliance and security review and are not yet publicly available as of May 2026.

No license information is mentioned in the explanation.

In plain English

SmartDirector is a research project from teams at the Chinese Academy of Sciences, Youku Moku-Lab, and Huazhong University of Science and Technology. It addresses a specific problem in AI video generation: most existing tools take a text description or a start and end frame, but give you little control over how the story inside the video develops or how the timing of scenes feels. SmartDirector aims to fix that by letting you specify multiple keyframes, images placed at particular points along the video, which the system then uses to build a coherent cinematic clip that respects the visual and narrative intent of each keyframe. The framework works in two stages. The first stage, called Director-Gen, produces a lower-resolution video that is conditioned on all the keyframes you provide. The second stage, called Director-SR, takes that rough output and sharpens it using the high-resolution versions of the keyframes as reference anchors, recovering fine visual detail in the final result. The system can handle a single continuous shot, a multi-shot sequence where scenes change, or extensions of an existing video clip. To train the models, the team built a data pipeline that extracts single-shot and multi-shot sequences from movies, covering both tightly framed scenes and longer narrative arcs. According to the paper, SmartDirector outperforms comparable methods in experiments. As of the project page launch in May 2026, the code and dataset are not yet publicly available. The README states they are pending a corporate compliance and security review before release. The repository currently serves as a paper announcement and citation reference. A preprint is available on arXiv under the identifier 2605.27891.

Copy-paste prompts

Prompt 1

Use SmartDirector's Director-Gen stage to generate a low-resolution video conditioned on three keyframes placed at the start, midpoint, and end of a 5-second clip, then feed the result into Director-SR to recover fine detail.

Prompt 2

Run the SmartDirector data pipeline to extract single-shot and multi-shot sequences from a movie and use them to fine-tune the keyframe-conditioned generation model.

Prompt 3

Reproduce the SmartDirector multi-shot benchmark results from the arXiv paper 2605.27891 on a held-out set of movie clips with custom keyframe placements.

Open on GitHub → Explain another repo

← orange-3dv-team on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.