dao-ailab/flash-attention

FlashAttention is a highly optimized implementation of the "attention" mechanism used in AI language models and other transformer-based neural networks. Attention is the core computation that lets an AI model figure out which parts of its input are relevant to each other, for example, which words in a sentence relate to which other words. Standard attention is notoriously slow and memory-hungry because it requires storing and processing a large matrix that grows quadratically with the length of the input. This becomes a serious bottleneck when working with long documents or large models. FlashAttention solves this by restructuring how the computation is done to minimize the number of times data needs to move between the GPU's fast on-chip memory and slower off-chip memory. This makes it both faster and significantly more memory-efficient, without changing the mathematical result, it computes the exact same answer as standard attention, just more efficiently. You would use FlashAttention when training or running large transformer models, especially when dealing with long sequences, and you want to reduce GPU memory usage and speed up training. It has become widely adopted in production AI systems. The library is written in Python and CUDA, requires PyTorch 2.2 or newer, and supports NVIDIA (Ampere, Ada, Hopper, Blackwell) and AMD (ROCm) GPUs. Versions 1 through 4 are available, each with further performance improvements.