senguptaumd/background-matting

★ 4,774PythonAudience · researcherComplexity · 4/5LicenseSetup · hard

Mindmap

mindmap
  root((background-matting))
    What It Does
      Cut subject from background
      No green screen needed
      Generate alpha matte
      Composite onto new scene
    How It Works
      Two-photo approach
      Neural network matting
      Segmentation setup step
    Inputs
      Subject photo or video
      Background reference shot
      Target new background
    Audience
      CV researchers
      Photographers
      Video editors

mindmap root((background-matting)) What It Does Cut subject from background No green screen needed Generate alpha matte Composite onto new scene How It Works Two-photo approach Neural network matting Segmentation setup step Inputs Subject photo or video Background reference shot Target new background Audience CV researchers Photographers Video editors

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Cut a person out of a portrait photo with natural hair edges by providing a matching clean background reference shot instead of a green screen.

USE CASE 2

Remove the background from a video clip and composite the subject onto a virtual scene without studio equipment.

USE CASE 3

Generate alpha mattes for a portrait dataset to use in training or evaluating other image segmentation models.

Tech stack

PythonPyTorchOpenCV

Getting it running

Difficulty · hard Time to first run · 1h+

Requires a GPU and a Python deep learning environment, also requires capturing a separate clean background reference photo for every scene.

Free for personal and research use only, commercial use is not permitted under the Creative Commons non-commercial license.

In plain English

This repository contains the code for a research project called Background Matting, published at the IEEE CVPR 2020 computer vision conference by researchers at the University of Washington. The project tackles a classic photography and video problem: how do you cleanly separate a person from their background so you can place them in front of a different scene? Traditional solutions use a green screen, which requires controlled studio lighting and a dedicated colored backdrop. Background Matting takes a different approach. Instead of a green screen, you capture two photos: one with the subject in frame, and one of the same background without the subject. The system compares these two images and uses a neural network to figure out precisely which pixels belong to the person, including difficult areas like hair, which would normally be hard to cut out cleanly. The output is called an alpha matte, which is a map that describes how transparent each pixel is at the boundary between the subject and background. With this matte, you can composite the subject onto any background you choose, with realistic edges that look natural rather than jagged or blurry. The code supports both photos and videos. For photos, you provide a subject image, a matching background image, and a target background. A setup step runs a segmentation model to roughly identify the person, then aligns the background image to the subject image to account for any camera movement between shots. Running the main script then produces the composited output. The repository also notes that a newer version of the project, Background Matting V2, has been released separately with better quality and real-time performance at high resolutions. The original version covered here is the initial research release and is intended for non-commercial use under a Creative Commons license.

Copy-paste prompts

Prompt 1

How do I run the Background Matting script to composite a person from subject.jpg onto a new background, given a reference image of the empty scene?

Prompt 2

What does the setup alignment step in Background Matting do, and how do I run it when my camera moved slightly between the subject shot and the background reference shot?

Prompt 3

How do I apply Background Matting to a video file frame by frame and what output format does the script produce?

Prompt 4

What is the difference between Background Matting V1 in this repo and Background Matting V2, and should I use V2 for a new project that needs real-time high-resolution output?

Open on GitHub → Explain another repo

← senguptaumd on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.