netflix/metaflow

★ 10,091PythonAudience · dataComplexity · 3/5Setup · moderate

Mindmap

mindmap
  root((Metaflow))
    What it does
      ML workflow framework
      Experiment tracking
      Cloud scaling
    Tech Stack
      Python
      AWS
      Azure
      GCP
    Use Cases
      Model training
      Pipeline deployment
      Experiment comparison
    Audience
      Data scientists
      ML engineers
      Research teams

mindmap root((Metaflow)) What it does ML workflow framework Experiment tracking Cloud scaling Tech Stack Python AWS Azure GCP Use Cases Model training Pipeline deployment Experiment comparison Audience Data scientists ML engineers Research teams

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Write a machine learning workflow once and have it run locally for fast testing, then automatically scale to cloud GPU machines for full training.

USE CASE 2

Track every experiment run, artifact, and model version so you can compare past results or reproduce any run exactly.

USE CASE 3

Deploy a finished ML pipeline to production with a single command so it runs on a schedule or reacts to external events.

USE CASE 4

Manage Python environment dependencies automatically so the same code runs identically on your laptop and in the cloud.

Tech stack

PythonAWSAzureGCP

Getting it running

Difficulty · moderate Time to first run · 30min

Requires a cloud account for remote execution, local mode works without cloud credentials for initial testing.

In plain English

Metaflow is a Python framework built at Netflix for developing and running AI and machine learning systems. It is designed to cover the full lifecycle of a project: starting from experimenting in a notebook on your laptop, scaling up to large parallel jobs on cloud computing clusters, and eventually deploying to a production scheduler that runs the workflow automatically. Today it is maintained by a company called Outerbounds and is used in production at organizations including Amazon, DoorDash, Goldman Sachs, and Ramp. Netflix itself runs over 3,000 AI and machine learning projects on it. The core idea is that you write Python code defining a workflow as a series of steps. Each step can run locally for fast testing or be pushed to remote cloud compute automatically, including GPU-enabled machines. Metaflow tracks every run, every artifact, and every model version so you can look back at past experiments or compare results across runs. It also manages dependencies so that the Python environment your code needs travels with it into the cloud. Deploying a finished workflow to production is done with a single command. Metaflow integrates with production-grade orchestration systems that handle scheduling and reliability, and it can react to external events to trigger new runs automatically. It works on AWS, Azure, and GCP. Installation is through pip or conda. The README points to a free interactive sandbox at outerbounds.com/sandbox as the quickest way to try it without local setup, and links to a tutorial that walks through creating your first workflow step by step. A Slack community is available for support.

Copy-paste prompts

Prompt 1

Show me how to write a Metaflow pipeline that trains a machine learning model locally and then scales to AWS for GPU training with minimal code changes.

Prompt 2

How do I use Metaflow to track experiment runs and compare model accuracy across different hyperparameter settings in the same project?

Prompt 3

Deploy my Metaflow workflow to run automatically on a schedule in AWS production. Walk me through the deployment steps and required configuration.

Prompt 4

How do I make sure the Python packages I use locally are automatically available when my Metaflow job runs on a remote cloud machine?

Prompt 5

Set up Metaflow to trigger a new training run automatically whenever new data arrives in an S3 bucket.

Open on GitHub → Explain another repo

← netflix on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.