sonthonaxrk/demandmap

Demandmap is a Rust based tool for working with very large numeric arrays stored in cloud storage, specifically Amazon S3 or S3 compatible services, without downloading the whole file first. Normally, to read a big array of numbers stored remotely, you have to download the entire file to your computer before you can use any of it. Demandmap avoids that by using a low level operating system feature called userfaulting, which lets the program pretend a large chunk of memory already exists and only fetch the real data from the network the moment your code actually tries to read a specific part of it. As you read through the array, Demandmap quietly downloads just the pieces you touch, in small chunks, and saves those chunks to a local cache file on disk. Repeated reads of the same data, or restarting the program later, are then fast because the pieces are already cached locally instead of being re-downloaded. The README shows an example using two large numeric files, each over 400 megabytes, being loaded and combined into a table with tens of millions of rows in about 100 milliseconds, because only the small pieces actually needed were pulled over the network. The project exposes both a Python interface, built with a tool called maturin that connects Rust code to Python, and a native Rust interface for programs written directly in Rust. The Python demo installs a few packages, builds the Rust extension, and runs a script that fetches and reads remote array data as if it were a normal local array. The author is upfront that this is an early, rough project, not ready for production use, and that it currently only works on macOS. Support for Linux and Windows, proper error and signal handling, and other caching options are listed as still unfinished.