masatoagi/ncd-memory

ncd-memory is a Rust library that gives AI agents a persistent external memory store, a way to remember and search through past conversations or notes, without needing a separate AI embedding model to do it. Most memory systems for AI agents rely on embedding models: neural networks that convert text into numeric vectors so that similar sentences end up near each other in a mathematical space. ncd-memory takes a different approach. It uses GZIP compression to measure how similar two pieces of text are. The idea comes from information theory: if two texts compress together much more efficiently than separately, they share structure and are probably semantically related. This measure is called Normalised Compression Distance (NCD), and it works with standard compression software, no trained model required. In practice, a developer adds ncd-memory as a dependency, opens a memory database, logs conversation turns, and calls a retrieve function with a query string to get back the most relevant past entries. The retrieval works in two stages: first a fast keyword search, then a GZIP-based semantic search for anything the keyword pass might miss. The library also supports user profile preferences that can be injected into prompts, and session tracking so memories from different conversations stay organised. An unusual feature is that the library's core logic is formally specified in Lean 4, a proof-assistant language used in mathematics research. Parts of the Rust implementation are verified to match the abstract specification, with the gaps documented openly. Built in Rust, stored in SQLite, licensed MIT.