facebookresearch/sparse-delta-memory

Sparse-Delta-Memory, or SDM, is Meta's research code released alongside an academic paper about a new way for AI language models to remember information across very long pieces of text. Most large language models struggle to recall details from far earlier in a long document or conversation, and this project proposes a new kind of memory component meant to help with exactly that problem. The core idea is a large, sparse memory bank that the model reads from and writes to using something called product keys, combined with a gating mechanism, similar in spirit to a technique called deltanet, that controls when the memory forgets old information and when it accepts new information. This is not an end user application. It is a research component meant to be plugged into a larger language model architecture, and the repository is built as a fork of Meta's existing Lingua research codebase, reusing its data preparation and training tools. The README includes benchmark results comparing this method against other memory approaches called Mamba2 and GDN, measured on perplexity, which is a measure of how well a model predicts text, and on a long context retrieval test called RULER. Across most of the reported tasks, SDM performs better than the comparison methods, especially at longer context lengths, though it does worse on a couple of individual tasks. Running this code requires real machine learning infrastructure: a modern GPU from Nvidia's Ampere or Hopper generation, a specific and fairly recent version of PyTorch and Triton, a working CUDA toolkit, and a compiler, since custom GPU kernels are compiled the first time they run. The README does provide a quick smoke test that exercises the GPU kernels without needing any real training data, plus instructions for a minimal training run using randomly generated text, before pointing to the steps needed for full scale training with real datasets. The code is released under a Creative Commons Attribution NonCommercial license, meaning it can be used and shared but not for commercial purposes.