midhunharikumar/ferroload

When you train a machine learning model on images, audio, or other mixed data, one of the bottlenecks is how fast you can load that data from disk or cloud storage into your training code. Ferroload is a data format and loading library built in Rust to address that bottleneck, with a Python interface for the people actually writing training code. The format stores data in sharded tar archives (essentially bundled file collections) and pairs them with an index file in a column-oriented format called Parquet. The Parquet index lets you query your dataset using SQL-style conditions, for example filtering to samples with a specific label or caption, without reading any of the actual image or audio data. That index can also be queried directly with a tool called DuckDB without installing Ferroload at all. Because the core decoding logic runs in Rust, Ferroload can do parallel image decoding inside a single process without needing multiple worker processes the way most Python data loaders do. Benchmarks in the repository compare it against two popular alternatives on standard image collections. Results show it loading small images roughly 1.8 times faster than Hugging Face datasets and about 3.9 times faster than WebDataset under equal conditions. For datasets stored in cloud object storage like Google Cloud Storage, the gap is larger because Ferroload coalesces remote reads rather than fetching small chunks one at a time. The Python package installs from PyPI with a single command and provides classes for writing datasets, reading them, filtering by query, and connecting to PyTorch training loops. Cloud storage including Amazon S3, Google Cloud Storage, and Azure is included in the prebuilt package. You can extend an existing dataset by adding computed columns or new modalities as additive layers without rewriting the original data, and those operations are designed to be resumable if interrupted. Building from source requires a Rust toolchain and a tool called maturin that compiles the Rust code into Python extension modules.