This is a rigorous explainer of the quadratic complexity of self-attention. Its central fact is that self-attention lets every one of n positions attend to all n positions, which is n squared interactions by definition, so attention costs order n squared times d in compute and order n squared in memory, and it cannot be made asymptotically cheaper without changing what attention does: attending to less (sparse), approximating the all-pairs interaction (linear), or scheduling it more cleverly (FlashAttention). It derives the compute from the matrix shapes: the scores Q times K transpose multiply an n by d matrix by a d by n matrix to make an n by n matrix whose n squared entries each cost about d multiply-adds, giving order n squared d; the softmax over the n by n matrix is order n squared; and the weighted values A times V multiply n by n by n by d for order n squared d again, so one attention layer is order n squared d, and multi-head does not change this because the h heads split the width d k equals d over h and h times n squared times d k equals n squared d. It contrasts this with the feed-forward network cost of order n times d squared, so attention overtakes the FFN once n exceeds d, with real constants around n approximately four d, which for d equals 4096 is about 16 thousand tokens, meaning long context is exactly the regime where the quadratic dominates. It then derives the order n squared memory from materializing the n by n attention matrix, and gives a concrete number: at n equals 128 thousand tokens in bf16 a single n by n matrix is about 34 gigabytes for one head one layer, exceeding an 80 gigabyte H100. It draws the crucial distinction that FlashAttention, from Dao and colleagues 2022, does not reduce the order n squared d compute but removes the order n squared memory by tiling and an online softmax in on-chip SRAM so the full matrix is never written to main memory, cutting the footprint to order n and slashing memory traffic; the quadratic compute remains, only the quadratic memory and its input-output cost are gone. It locates where the quadratic bites in inference: prefill, reading the whole prompt in one parallel pass, is compute-bound and quadratic so time to first token grows like n squared; decode, generating one token at a time attending to the growing KV cache of size two n d L, is memory-bandwidth-bound, and summed over a length n generation the KV traffic is again order n squared. It then reads the entire efficient-attention literature as four strategies against the n squared: attend to less (Sparse Transformer, Longformer, BigBird, sliding-window), approximate the all-pairs interaction with kernels or low rank for genuine order n (Linformer, Performer, linear transformers, Kimi Delta Attention), schedule it better with exact IO-aware kernels (FlashAttention, Flash-Decoding), and shrink the decode state by sharing or compressing keys and values (Multi-Query and Grouped-Query Attention, Multi-head Latent Attention), plus the radical branch that abandons softmax attention for state-space recurrences with fixed state (Mamba). The through-line reaches Kimi K3, Moonshot’s open 2.8 trillion parameter model built on the linear-attention Kimi Delta Attention. Caveats: sparse and linear attention trade exactness for scale and can miss long-range dependencies, FlashAttention does not move the asymptote, linear and state-space models compress history into a fixed state that cannot store unbounded detail, and the numbers are order of magnitude. Two interactive widgets let the reader drag the context length and watch attention compute, the materialized attention matrix memory, and the KV cache grow against the FFN and an 80 gigabyte GPU, and filter the efficiency-method lineage by which cost each method attacks.