apache/amoro

Apache Amoro is a management system for data lakes, the large storage systems where companies keep massive amounts of raw data. Think of it as a smart layer that sits on top of your data lake and automatically keeps things tidy, so your queries run fast and your storage costs stay low. Instead of manually fixing fragmented files or cleaning up old data, Amoro handles that maintenance for you. At its core, Amoro provides a central management service that connects your data storage with the processing engines you already use, like Flink, Spark, and Trino. It continuously optimizes tables by merging small files, removing duplicates, sorting data, and deleting expired files. It also provides a unified catalog, meaning all your different tools can look in one place to find table information. Amoro works with several open data formats, including Iceberg, Paimon, and Hive, so you can choose the format that fits your needs without being locked into one option. This project is aimed at data teams and platform engineers who manage large-scale data infrastructure. For example, a company processing real-time event data might use Amoro to keep their Iceberg tables optimized automatically, ensuring analytics queries stay fast without someone writing cleanup scripts. Teams migrating from traditional Hive setups can use the Mixed-Hive format to get data lake benefits while keeping existing Hive-based workflows intact. It is designed for organizations that want an "out-of-the-box" data warehouse experience on top of open storage formats, whether deployed on private servers, in the cloud, or across multiple clouds. One notable design choice is the plugin architecture. Amoro separates its management service from the engines doing the actual optimization work, so you can plug in different optimizers or even use external systems like Kafka for real-time data processing. It also includes a web dashboard and SQL command-line tools, making it accessible beyond just engineers writing code.