redai-infra/hypic

HYPIC is a research project describing a new technique for serving large language models faster, specifically ones that mix two different internal designs called linear attention and full attention, known together as hybrid attention models. The README is short and mainly points to an academic paper rather than working code, so this repository currently documents an idea and its results rather than something you can run today. The problem HYPIC addresses is caching. Many AI serving systems speed up repeated or similar requests by reusing previously computed data instead of recalculating it from scratch. That trick already works well for regular attention models, but hybrid models mix in a different internal state that does not carry the same kind of reusable, position independent information, so the usual caching tricks do not transfer over. HYPIC introduces a way to make that caching work for hybrid models too. It does this through three ideas described in the README: caching how a chunk of the model's internal state changes over time so it can be reused instead of recomputed, recomputing a small window of data at the start of each chunk to fix accuracy at the seams where chunks join, and splitting a request's uncached portions across multiple workers so they can be processed at the same time. Together, according to the paper, these bring large speed improvements to how quickly a language model starts responding and how many requests it can handle per second, compared to an existing caching approach, while getting close to the speed of never caching anything and always recomputing fully. HYPIC is built on top of SGLang, an existing open source system for serving language models. As of this README, the authors say the code itself has not been released yet and invite people to star or watch the repository for updates. This project is aimed at researchers and engineers who build or operate large language model serving infrastructure and care about response speed at scale, rather than everyday app users.