prefecthq/prefect

Prefect is a Python framework for orchestrating data pipelines, for taking a script that pulls, transforms or moves data and turning it into a workflow that runs reliably on a schedule, recovers from failures, and can be monitored. Its own description calls it "the simplest way to elevate a script into a production workflow," for building resilient, dynamic data pipelines that react to the world around them. The way you use Prefect is to decorate ordinary Python functions: a small unit of work becomes a "task" with the @task decorator, and a function that strings tasks together becomes a "flow" with the @flow decorator. Once your code is structured that way, Prefect gives it features like scheduling, caching of expensive steps, automatic retries when something fails, dependencies between steps, branching logic, and event-based automations that can trigger workflows in response to outside events. You can run a self-hosted Prefect server locally, its UI lives at http://localhost:4200, to watch flow runs and inspect what happened, or use the managed Prefect Cloud dashboard for the same observability without running your own infrastructure. A "deployment" packages a flow with a schedule (such as a cron expression) so it runs on its own. You would reach for Prefect when a data script has outgrown being run by hand or by a fragile cron job, for example an ETL job, a regular report, a machine-learning training or inference pipeline, or any sequence of API calls and data transformations that needs retries, scheduling and a UI to debug failures. Installation is via pip or uv, and Prefect requires Python 3.10 or newer.