explaingit

spotify/luigi

📈 Trending★ 18,721PythonAudience · dataComplexity · 3/5ActiveLicenseSetup · moderate

TLDR

Python library for building and running automated data pipelines where tasks depend on each other; handles scheduling, retries, and skipping completed work.

Mindmap

mindmap
  root((Luigi))
    What it does
      Task orchestration
      Dependency management
      Automatic scheduling
    Key features
      Web dashboard
      Retry logic
      Skip completed tasks
    Use cases
      Data processing
      ML training jobs
      ETL workflows
    Tech stack
      Python
      Hadoop
      Spark
    Audience
      Data engineers
      Pipeline builders

Things people build with this

USE CASE 1

Build multi-step data processing pipelines that run daily or weekly, automatically skipping steps already completed.

USE CASE 2

Orchestrate machine learning training jobs that depend on data preparation, feature engineering, and model evaluation steps.

USE CASE 3

Create ETL workflows that extract data from databases, transform it, and load results into data warehouses or dashboards.

USE CASE 4

Manage complex Spark or Hadoop jobs where later tasks depend on outputs from earlier computations.

Tech stack

PythonHadoopHivePigSpark

Getting it running

Difficulty · moderate Time to first run · 30min

Requires Python environment setup and understanding of task dependency configuration; Hadoop/Spark integration optional but adds complexity.

Use freely for any purpose, including commercial use, as long as you keep the copyright notice and license text.

In plain English

Luigi is a Python library for building and managing automated pipelines, sequences of tasks that need to run in a specific order, where each step depends on the results of previous ones. Think of it like a makefile for long-running data jobs: you describe what each task needs as input and what it produces as output, and Luigi handles running everything in the right order, skipping tasks that are already done, and retrying or reporting failures. It was originally developed at Spotify and used internally to run thousands of tasks every day, including machine learning jobs, data exports, and internal dashboards. The library is particularly suited for workflows that take hours or days to complete and involve many interdependent steps, such as processing large datasets or training models. Luigi comes with support for common data infrastructure including Hadoop, Hive, Pig, and Spark jobs, as well as database operations. Every piece of logic, including the dependency graph, is written in plain Python rather than configuration files or domain-specific languages, which makes it easy to express complex dependencies like date-based calculations. A web interface is included for searching and visualizing the dependency graph and task statuses. Luigi is installed via pip.

Copy-paste prompts

Prompt 1
Show me how to write a Luigi task that depends on another task's output, with a concrete example of reading and writing files.
Prompt 2
How do I use Luigi to run a multi-step data pipeline where each step only runs if the previous one succeeded?
Prompt 3
Write a Luigi pipeline that processes CSV files in parallel and aggregates the results into a final report.
Prompt 4
How do I visualize my Luigi task dependencies and monitor which tasks have completed using the web interface?
Prompt 5
Create a Luigi task that retries automatically if it fails, with exponential backoff between attempts.
Open on GitHub → Explain another repo

Generated 2026-05-18 · Model: sonnet-4-6 · Verify against the repo before relying on details.