yifei-zuo/parallax

Parallax is a Python library that implements a new form of attention mechanism for AI language models, proposed in a 2026 research paper. Attention mechanisms are the core mathematical operation that allows AI models to decide which parts of text to pay attention to when generating or processing language. Parallax offers a modified version called Parameterized Local Linear Attention (LLA) that the authors claim has theoretical advantages over the approach most large models use today. Unlike some alternatives that trade accuracy for speed by reducing computations, Parallax keeps the same computational structure as standard attention. It still uses a KV cache during text generation, which is the memory that stores previous context. What it changes is the mathematical formula used to score which words relate to which other words. The README notes this is not a linear-complexity mechanism and should not be confused with approaches that simplify the math at the cost of accuracy. The library includes two main components: a training kernel built with Triton, a GPU programming tool, and a decoding kernel using CuTeDSL. Benchmark results in the README compare timing against two widely used alternatives called FA2 and FA3. For small batches, Parallax is faster across most tested configurations. For large batches, it roughly matches FA3. Installation requires Python and a compatible NVIDIA GPU. The README notes that the optimized kernels are tested on NVIDIA Hopper GPUs specifically, but a plain PyTorch version is included for correctness testing on other hardware. The repository also includes benchmarking scripts so researchers can reproduce the speed comparisons themselves. The intended audience is machine learning researchers and engineers who build or study transformer-based AI models and want to experiment with alternative attention formulations. The README links to an arXiv preprint for the underlying theory.