This is a mechanics-deep explainer of FlashAttention by Dao and colleagues 2022. Its central idea is that attention’s wall-clock bottleneck is not the FLOPs but moving the n by n attention matrix to and from slow GPU memory, so the fix is to keep the computation in fast on-chip memory, never write the matrix out, and reconstruct softmax on the fly, giving the same exact answer with the same FLOPs but multiples faster. A GPU has two memories: SRAM, on-chip, tens of megabytes, about 19 terabytes per second, where compute happens; and HBM, high-bandwidth memory, tens of gigabytes, a couple terabytes per second, about ten times slower. Standard attention computes S equals Q K transpose, an n by n matrix, writes it to HBM, reads it back to softmax, writes the result, reads it again to multiply by V, so the giant matrix crosses the slow HBM boundary several times; the order n squared arithmetic is fine, it is the order n squared memory traffic that dominates, so attention is memory-bound. FlashAttention is IO-aware: it splits Q, K, V into blocks that fit in SRAM, computes attention block by block entirely on chip, and never writes the n by n matrix to HBM at all. Tiling is easy but the softmax is hard because to normalize row i you need its denominator, a sum over all keys, and the row max for numerical stability, neither of which you know until you have seen every block. The online softmax solves this by keeping three running quantities as blocks stream in: the running max m, the running normalizer ell equal to the sum of exp of s minus m, and the running output accumulator O; when a new block arrives with its own local max, update the max to m new equals max of m and the block max, then rescale ell and O by the correction factor exp of m minus m new which retroactively fixes earlier blocks as if they had known the true max, and add the new block’s contributions using exp of s minus m new. After the last block O over ell is bit for bit the softmax-weighted output, no approximation. The FLOPs are essentially unchanged at order n squared d, but the HBM traffic collapses from writing and re-reading an order n squared matrix to moving order n data, so the number of slow HBM accesses drops by a large factor tied to SRAM size, and because attention was memory-bound this cuts wall-clock time, 2 to 4 times faster in practice and exact, with HBM accesses order n squared d squared over M where M is SRAM size. The backward pass would reintroduce order n squared memory if it stored the matrix, so FlashAttention recomputes needed blocks on the fly in SRAM, trading nearly-free extra FLOPs for decisive memory savings. FlashAttention-2 in 2023 reworked work partitioning across warps and thread blocks to roughly double throughput, and FlashAttention-3 in 2024 exploits Hopper GPU asynchrony and FP8. Because it is exact it won: sparse and linear attention accept approximation for asymptotic savings, but FlashAttention gives the full attention result computed IO-efficiently, removing the reason to approximate at moderate to long contexts, so nearly every model runs it; but it does not change the asymptote, compute is still order n squared d, so at extreme context genuinely sub-quadratic methods still win, and it is a kernel not an architecture so you get the speedup without changing the model. In the lineage FlashAttention is the reschedule-it branch that does not approximate, and paired with grouped-query attention it defines the attention stack of essentially every modern model, with the frontier pushing past it at million-token contexts where the constant factor stops being enough and sub-quadratic and state-space lines take over. Two interactive widgets let the reader step a streaming softmax over four blocks and watch the running max, normalizer, and output converge to the exact softmax times values without materializing the full row, and compare standard order n squared HBM traffic against FlashAttention’s order n traffic as sequence length grows with identical FLOPs.