yogthos/flatiron

Flatiron is a Clojure library for running fast analytical queries on data that lives in memory. It is aimed at Clojure programs that need to do grouping, filtering, sorting, and aggregation on reasonably large tables without pulling in a full embedded database. The key design choice is how it stores data. Standard Clojure code typically represents a table as a list of maps, one map per row. That works fine for a few thousand records but gets slow on larger datasets because each row is a separate object in memory with a lot of overhead. Flatiron instead stores each column as a plain Java primitive array: integers go into a long array, floating-point numbers into a double array, and so on. This means the computer can loop through the data without chasing pointers, and the JVM can optimize the inner loops into tight native code. The library processes data in batches of 1024 rows to reduce the cost of figuring out which operation to use on each pass. The query interface is a small set of functions and macros that look similar to SQL. You can filter rows with conditions (greater than, equals, and so on), group by one or more columns, compute aggregations like sum, average, count, minimum, and maximum, and create pivot tables. You can also run graph algorithms like PageRank on the same in-memory tables. The library has no external dependencies beyond one standard Clojure concurrency library and requires JDK 18 or later. Date and time values are stored as plain numbers under the hood (for example, a date is stored as the number of days since a fixed starting point) so they sort and compare correctly without creating extra objects. You can also define your own custom column types with the same approach. Flatiron is intended as a lightweight alternative to pulling in an embedded database when you just need to analyze some in-memory data in a Clojure project. It is installed via a git dependency entry in a Clojure project file.