landisen/flashmorph

FlashMorph is a research project from a team at Fudan University, ByteDance Seed, and The Chinese University of Hong Kong, describing a paper called Morphing into Hybrid Attention Models. It addresses a technical problem in large language models: the standard attention mechanism these models use to process text becomes slow and memory hungry as the amount of text grows. Some newer models use a faster alternative called linear attention, but pure linear attention models can lose some of the reasoning strengths of standard attention. FlashMorph is a method for taking an already trained standard Transformer model and converting a chosen set of its layers into a faster linear attention style, while keeping the rest of the layers as normal full attention, creating what the authors call a hybrid attention model. The key contribution described is an efficient way to decide which layers should be converted, working within a fixed budget for how many layers are allowed to stay as full attention. The README reports several results from the paper. On a long context retrieval test called Needle in a Haystack, the converted models perform strongly while only needing 20 million tokens worth of computation to decide on the layer conversion, which the authors describe as a small cost. The resulting hybrid models are said to keep their commonsense reasoning ability and do better on tasks that require recalling specific information, compared across different linear attention backbones. The hybrid architecture is also reported to process long text faster and use less GPU memory than a standard full attention Transformer, and the method for choosing which layers to convert is described as much cheaper to run than earlier approaches. This repository does not contain runnable source code, only the README, images of result tables, and a link to the paper on arXiv, along with a citation entry and a contact email for questions. It would mainly interest machine learning researchers and engineers working on making large language models faster and cheaper to run on long documents.