pola-rs/polars

Polars is a high-performance data analysis library for working with tabular data, tables of rows and columns, like a spreadsheet or a database table. It solves the same problem as Python's pandas library but is designed from the ground up for speed and efficiency. Polars is written in Rust, which gives it a significant performance advantage, and it exposes APIs in Python, Rust, Node.js, and R so you can use it from whichever language you prefer. The key concepts behind Polars's performance are several. It stores data in a columnar format based on the Apache Arrow standard, meaning all values in a column are packed together in memory, which is much faster to process than row-by-row storage. It uses multiple CPU cores in parallel and takes advantage of SIMD (Single Instruction Multiple Data) instructions, which let a single CPU operation process multiple values at once. Polars also has a lazy execution mode: instead of running each operation immediately, it builds up a query plan first and then optimizes it before running, similar to how a database query planner works. It can even process datasets larger than your available RAM by streaming the data in chunks. You would use Polars when working with large datasets in Python (or Rust/Node.js/R) and finding pandas too slow or too memory-hungry. Data scientists, analysts, and engineers handling gigabytes of CSV files, Parquet data, or database exports benefit most from it. Installing it is as simple as running pip install polars, and it requires no external C dependencies. The core library is written in Rust, with Python bindings generated by the PyO3 library.