lakehq/sail

Sail is a drop-in replacement for Apache Spark, the widely used data processing engine. If your team already runs Spark to power data pipelines, analytics, or machine learning workloads, Sail lets you swap out the underlying engine without rewriting any of your existing code. The pitch is simple: it claims to be roughly four times faster and 94% cheaper to run than Spark, while being fully compatible with Spark SQL and the Spark DataFrame API. The way it works is straightforward from a user's perspective. You install Sail as a Python package, start the Sail server, and then point your existing PySpark code at it. Your code doesn't need to change at all. Under the hood, the project is written in Rust rather than running on the Java Virtual Machine (JVM) like Spark does. This matters because the JVM introduces memory overhead and periodic pauses for cleanup that can slow things down. By using Rust, Sail avoids those issues, resulting in faster startup times, lower memory usage, and more predictable performance. This tool is designed for data engineers, data scientists, and engineering teams who are already working with Spark and want to cut their infrastructure costs or speed up their queries. For example, if your company runs nightly batch jobs to transform large datasets, or if you have streaming pipelines that need to process data continuously, Sail aims to handle those workloads more efficiently. It supports popular data lake formats like Delta Lake and Apache Iceberg, and connects to storage systems including AWS S3, Google Cloud Storage, and Azure, so it fits into existing cloud data stacks. One notable aspect of the project is its claim of zero-copy execution for Python user-defined functions (UDFs). In Spark, running Python code alongside the main engine often requires moving data back and forth, which creates overhead. Sail shares data directly with Python using in-memory pointers, which can make custom Python logic run much faster. The project also includes a compatibility check script that scans your existing codebase to flag any Spark functions that might not yet be fully supported, helping teams assess migration risk before making the switch.