ion-elgreco/rivers

Rivers is an orchestration platform for data and machine learning pipelines. Orchestration means coordinating when and in what order your data processing steps run, for example pulling raw data, then transforming it, then writing the result, while handling dependencies, scheduling, and failures automatically. You define your pipeline in Python by decorating ordinary functions with an Asset decorator, and Rivers figures out the execution order by resolving the dependency graph automatically. Under the hood, graph resolution, execution planning, and scheduling all run in compiled Rust, so the control plane stays fast even on graphs with thousands of nodes. Python is only the interface you write against, it never runs on the scheduler itself. Rivers supports several asset types: single outputs, multi output, sub pipeline graphs, and external assets. It handles partitioning, slicing your data by time windows such as daily, hourly, or custom cron, by multiple dimensions, or by dynamic partitions defined at runtime. Built in storage handlers include in memory, pickle based object stores, and Delta Lake with merge support. For execution, you can run steps concurrently with parallel subprocess workers or deploy each step as its own pod on Kubernetes using the included operator and custom resource definitions. The single binary development experience boots the embedded database, scheduler, and web interface together in one command. Production deployments connect to a SurrealDB server backed by RocksDB instead. Rivers suits you when you need a pipeline tool light enough to run locally but structured enough to scale to Kubernetes without rewriting your definitions.