dao-ailab/flash-attention

📈 Trending★ 23,831PythonAudience · developerComplexity · 4/5ActiveLicenseSetup · hard

Mindmap

mindmap
  root((repo))
    What it does
      Optimizes attention computation
      Reduces GPU memory usage
      Speeds up training
    How it works
      Minimizes data movement
      Uses on-chip memory
      Exact same math result
    Use cases
      Training large models
      Processing long documents
      Running transformer networks
    Tech stack
      Python
      CUDA
      PyTorch
    Supported hardware
      NVIDIA GPUs
      AMD ROCm GPUs
      Multiple architectures

mindmap root((repo)) What it does Optimizes attention computation Reduces GPU memory usage Speeds up training How it works Minimizes data movement Uses on-chip memory Exact same math result Use cases Training large models Processing long documents Running transformer networks Tech stack Python CUDA PyTorch Supported hardware NVIDIA GPUs AMD ROCm GPUs Multiple architectures

Things people build with this

USE CASE 1

Train large transformer models faster with lower GPU memory requirements.

USE CASE 2

Process long documents or sequences without running out of GPU memory.

USE CASE 3

Speed up inference on language models and other attention-based neural networks.

USE CASE 4

Integrate optimized attention into custom AI model architectures.

Tech stack

PythonCUDAPyTorchNVIDIAAMD ROCm

Getting it running

Difficulty · hard Time to first run · 1h+

Requires CUDA/ROCm GPU, PyTorch with GPU support, and C++ compilation of custom kernels.

Use freely for any purpose including commercial. Keep the copyright notice and don't use the authors' names to endorse derivative work.

In plain English

FlashAttention is a highly optimized implementation of the "attention" mechanism used in AI language models and other transformer-based neural networks. Attention is the core computation that lets an AI model figure out which parts of its input are relevant to each other, for example, which words in a sentence relate to which other words. Standard attention is notoriously slow and memory-hungry because it requires storing and processing a large matrix that grows quadratically with the length of the input. This becomes a serious bottleneck when working with long documents or large models. FlashAttention solves this by restructuring how the computation is done to minimize the number of times data needs to move between the GPU's fast on-chip memory and slower off-chip memory. This makes it both faster and significantly more memory-efficient, without changing the mathematical result, it computes the exact same answer as standard attention, just more efficiently. You would use FlashAttention when training or running large transformer models, especially when dealing with long sequences, and you want to reduce GPU memory usage and speed up training. It has become widely adopted in production AI systems. The library is written in Python and CUDA, requires PyTorch 2.2 or newer, and supports NVIDIA (Ampere, Ada, Hopper, Blackwell) and AMD (ROCm) GPUs. Versions 1 through 4 are available, each with further performance improvements.

Copy-paste prompts

Prompt 1

How do I integrate FlashAttention into my PyTorch transformer model to reduce memory usage?

Prompt 2

Show me how to install FlashAttention and use it with a Hugging Face language model.

Prompt 3

What GPU architectures does FlashAttention support, and how do I check if mine is compatible?

Prompt 4

Explain the performance difference between standard attention and FlashAttention on a long-sequence task.

Prompt 5

How do I benchmark FlashAttention to measure speedup and memory savings on my model?

Open on GitHub → Explain another repo

Generated 2026-05-18 · Model: sonnet-4-6 · Verify against the repo before relying on details.