lordxmen2k/sparse-spectral-encoding

This project is a proof of concept for a way to shrink the storage size of embedding vectors, the long lists of numbers that AI systems use to represent the meaning of text so it can be searched by similarity. The technique, called Sparse Spectral Encoding, keeps only the strongest frequency components of each vector after a mathematical transform, then compresses those remaining values further before storing them. The idea is aimed at what the author calls the cold tier of memory for long-running AI assistants, meaning the huge, rarely accessed backlog of old conversation data that needs to be stored cheaply, searched reasonably fast, and can tolerate a small drop in accuracy compared to keeping every vector in full detail. The repository includes the encoder itself, a retrieval system built to work with the compressed format, and a benchmark that compares this method against four common alternatives such as basic rounding, a binary yes or no encoding, and a technique called product quantization. Tests are run first on generated synthetic data, with additional evaluation scripts for two established retrieval benchmark suites so the method can also be tested on real text embeddings rather than only made up numbers. The author is explicit that this is a work in progress, not a finished production library and not a published research paper. The core search logic runs in plain Python and NumPy rather than fast lower-level code, so it is meant to prove the math works rather than to be fast enough for real deployment as is. It optionally supports a Python speed-up library for parts of the search step. This is aimed at researchers and engineers working on AI memory or vector search systems who want to evaluate whether this compression idea holds up, not at general software users.