nohwai-software/raytention

RayTention is a research project that proposes a different way for AI language models to process text. Standard language models keep growing amounts of memory as they read longer pieces of text. The memory they store, called a KV cache, can reach hundreds of gigabytes for very long documents. RayTention is designed to replace that growing memory with a fixed, tiny summary that stays the same size no matter how long the text is. The core idea is to compress the entire history of a conversation or document into seven signals totaling 642 values, instead of storing every word's representation in full. These signals capture things like: what is the overall topic of what was just read, what is the most recently seen token, what single piece of context is most relevant right now, and how focused or scattered the model's attention is. A small neural network then processes these signals to produce the model's output. The practical result is that RayTention uses roughly 102 megabytes of GPU memory for inference at any context length, from 16,000 tokens to 1 million. Standard approaches need over 500 gigabytes for the same 1 million token context. According to benchmarks in the repo, RayTention matches standard attention in text prediction quality after 2,000 training steps, while using a fraction of the memory. It is slower in the current Python prototype because it lacks the hardware-optimized kernels that standard attention benefits from, but the authors note a Rust and CUDA version already reaches much higher speeds. The project is written in Python and uses PyTorch. Running the benchmark requires a CUDA-capable GPU and Python 3.10 or later. The repo includes a single benchmark script that trains both a standard transformer and a RayTention model on the same data, then compares memory usage and speed. A native CUDA kernel is listed as future work. The license is AGPL-3.0, a copyleft license that requires derivative works to be released under the same terms. The architecture is also subject to a pending U.S. patent application.