windyrobin/datafusion

Apache DataFusion is a fast, flexible SQL query engine written in Rust that lets you ask questions of your data, similar to what you'd do in a spreadsheet or database, but with much better performance and the ability to customize it for your specific needs. Instead of being locked into a single database product, DataFusion is a building block that developers and companies use to create their own data tools, pipelines, and query systems. Here's what you can do with it: write SQL queries or use a Dataframe API (a programmatic way to manipulate data) to search, filter, and analyze files like CSVs, JSON, Parquet, and Avro formats. DataFusion handles all the heavy lifting of parsing your query, figuring out the most efficient way to run it, and returning results. You can also use it from Python if you prefer, not just from Rust. The README mentions it performs competitively on benchmark tests, which matters if you're processing large datasets. Who uses this? Data engineers building custom analytics platforms, companies creating specialized database engines, and anyone writing data pipeline tools benefits from starting with a proven, open-source query engine instead of building one from scratch. Rather than being a consumer product like a traditional database, DataFusion is infrastructure, a foundation that other projects are built on top of. What's notable is the emphasis on customization. The project is designed to let you pick and choose which features you need (support for different file formats, encryption functions, date operations, etc.) so you don't pay overhead for capabilities you don't use. It's also part of the Apache Arrow ecosystem, which is a widely adopted standard for how data is organized in memory, making it compatible with many other data tools in the modern analytics stack.