Attention

Queries, keys and values

Let X contain one hidden-state row per token. Learned projection matrices produce three activation arrays: Q = XW_Q, K = XW_K, and V = XW_V. The matrices W are parameters. Q, K and V depend on the current input.

A query describes what a position needs. A key describes how another position can be found. A value carries the content returned from that position. This retrieval interpretation is useful, but these roles are learned rather than predefined semantic labels.

The dot product depends on vector norms as well as their angle. A high score does not establish a human-readable relationship. Softmax converts the allowed scores into nonnegative weights that sum to one. The output is therefore a weighted average of the values.

Multiple heads use separate query projections and can read different relations. Their outputs are concatenated and projected by W_O. This output projection mixes head information; it is not another attention distribution.

Token embeddings are not identical across the vocabulary. Repeated instances of the same token ID initially share the lookup vector, but position and earlier contextual layers can distinguish their hidden states. Attention Is All You Need ↗

How heads write to the residual stream

Reading values is not the final update to the token state. Each head returns a vector of width d_v. The output projection maps these vectors back to model width. The residual connection adds the result to the current state.

A projection can map a 768-wide token state to a 128-wide query. In a conventional 768-wide model with 12 equal-width heads, each head instead has width 64. The quotient is a layout choice, not a rule for every architecture.

Concatenation followed by a linear projection therefore equals a sum of projected head outputs. Heads need not write in the same direction. For one head, W_V,hW_O,h has rank at most d_v. This factorization concerns the value-to-residual map. The entire block is not one fixed low-rank linear map: P_h depends nonlinearly on X.

For a phrase such as “fluffy blue creature,” attention can transfer adjective information into the creature token's state. This is a conceptual interpretation, not proof that a particular head encodes adjectives. The updated state remains a learned vector, not a dictionary definition or necessarily a static word embedding.

Vector analogies can illustrate learned directions, but directions and semantic relations need not agree across models or contexts. Stacking layers allows information already contextualized at one position to influence another. Residual layout and normalization belong to Transformer Architecture.

Why divide by √d?

For independent, zero-mean, unit-variance query and key components, each product $q_i k_i$ has variance one. Summing d_k independent products gives variance d_k. Division by $\sqrt{d_k}$ restores variance one.

Without scale control, wider heads can create larger logits. Softmax can then concentrate almost entirely on one key. Its derivatives shrink near that saturated state, making some changes difficult to learn.

Head width is not the only variable. Increasing the number of visible keys changes the score distribution’s extremes and the attention entropy. A fixed variance scale does not guarantee fixed entropy across context lengths.

Su Jianlin studies entropy-oriented scales such as $\kappa\frac{\log n}{d_k}$ times $QK^\top$. This is a different objective from the standard variance argument. For causal attention, n is the active row length. Treat these scales as hypotheses to evaluate, not universal replacements.

Entropy and attention scaling ↗ · Gradient-based scale analysis ↗ · Key normalization ↗

Variance, temperature and normalization

The variance argument can be derived without assuming Gaussian components. For mutually independent zero-mean $q_i$, $k_i$ with variances $\sigma_q^2$, $\sigma_k^2$, $\mathbb E[q_i k_i]=0$ and $\mathbb E[q_i^2 k_i^2]=\sigma_q^2\sigma_k^2$.

Numerical overflow and attention saturation are different problems. Subtracting the row maximum makes exponentiation numerically safer; it leaves softmax probabilities unchanged. Dividing by a scale changes relative logits and therefore changes the probabilities.

For temperature T>0, $p=\operatorname{softmax}(s/T)$. Smaller T increases concentration for unequal scores; larger T approaches uniform weights. Equal logits remain uniform at every positive temperature. No temperature can create a preference when every score is identical.

At uniform weights over n keys, the Jacobian diagonal is $\frac{n-1}{n^2}$ and each off-diagonal is $-\frac{1}{n^2}$. These entries are not zero. Whether an objective trains well also depends on upstream derivatives, values and the loss. There is no universal optimal scale obtained by maximizing one Jacobian entry.

Entropy uses $H(p)=-\sum_j p_j\log p_j$, in nats for natural logarithms. A uniform row has $H=\log n$. Compare the number of allowed keys, not the padded batch length. A causal row near the start has fewer keys than a row near the end.

QueryNorm uses $\frac{q}{\|q\|_2}$; KeyNorm uses $\frac{k}{\|k\|_2}$. Only normalizing both produces a cosine score. QueryNorm alone still permits key norms to affect scores. KeyNorm alone still permits query norms to control scale. Zero-length vectors need an explicit epsilon or rejection policy.

Variance control, entropy across lengths and retrieval quality are separate objectives. The inherited Su analyses propose length-dependent scales and KeyNorm experiments; their findings are not universal replacements for $\sqrt{d_k}$. Measure entropy, gradients and held-out task quality together.

One masked attention row

Consider the third position of a four-token sequence. Its raw query–key products are [4, 1, −1, 3]. With d_k = 4, the scaled scores are [2, 0.5, −0.5, 1.5]. The fourth key is in the future, so causal attention excludes it before normalization.

The future token has exactly zero weight. Zeroing its probability after an unmasked softmax would leave the other weights summing below one. Apply the mask to logits first, or explicitly renormalize a mathematically equivalent implementation.

# q: [nq, dk], k: [nk, dk], v: [nk, dv]
# allowed: [nq, nk]; each query has at least one allowed key
scores = (q.float() @ k.float().T) / math.sqrt(q.shape[-1])
scores = scores.masked_fill(~allowed, -float("inf"))
weights = scores.softmax(dim=-1)
out = weights @ v.float()
# Teaching reference, not a fused production kernel.

Prefill, decode and the cache

Prefill processes many prompt queries together. Ordinary dense attention computes a quadratic set of query–key interactions. During autoregressive decode, a new query reads the existing prefix. The model can reuse earlier projected keys and values because those causal states do not change.

At prefix length n, one new query still reads roughly n keys and values. Caching removes repeated prefix projections and hidden-state computation; it does not make full-context retrieval constant-time. Over a growing generation, the cumulative retrieval work remains quadratic in generated length.

For 32 layers, 32 query heads, d_k = d_v = 128 and 8,192 positions, MHA stores 4 GiB in BF16 for batch one. Eight KV heads reduce this to 1 GiB. One KV head reduces it to 128 MiB. This example holds width, length and layer count fixed; it is not a throughput benchmark.

The cache design affects both capacity and decode bandwidth. A model that fits can still decode slowly if each step transfers too much cached state. Prefill and decode therefore need separate measurements. KV cache and serving →

MHA → MQA → GQA

Multi-head attention (MHA) gives each query head its own keys and values. This is flexible, but every position stores all KV heads. Multi-query attention (MQA, 2019) retains many query heads while sharing one KV head. The decode query still differs by head, but the retrieval database is shared.

Grouped-query attention (GQA, 2023) uses several KV groups. Each query head reads its assigned group. GQA sits between full independence and complete sharing; MQA is its one-group limit.

Head sharing can restrict representational freedom. The quality–cache tradeoff depends on training, tasks and head dimensions. Do not infer a guaranteed quality ranking from group count. Also do not explicitly replicate shared KV tensors in memory and then expect the intended savings.

MQA: Fast Transformer Decoding ↗ · GQA ↗

MLA: compress the cached representation

Head sharing changes how many KV groups exist. Multi-head latent attention (MLA, DeepSeek-V2, 2024) instead stores a shared compressed representation c. Learned maps recover richer per-head keys and values. The useful inference trick avoids materializing those expanded cached tensors.

Absorb W_k^T into the query projection. The query can now score cached c directly. Likewise, move the value expansion into the output projection after the weighted latent sum. This is an algebraic rearrangement under the stated conditions, not an approximation from dropping keys.

Expand once after reading, not once per cached token

Each head has its own weights and latent read. Its value expansion and output block are linear, so their product can follow the weighted sum.

Use two cached row vectors and one head. Fix its attention weights at 0.25 and 0.75. The expanded path forms both values before averaging. The latent path averages the cached vectors first, then applies the fused value/output map.

Both paths return the same vector. Only the placement of the linear maps changes. In a long cache, the latent path avoids expanding every stored token for this read. Scroll the comparison horizontally on a narrow screen.

Keep the same masks, weights and parameters in both paths. The displayed identities omit biases and dropout. A nonlinear operation on expanded keys or values generally prevents this simple absorption. Finite-precision multiplication order can change rounding; compare numerical tolerances, not bitwise equality.

Latent caching saves expanded KV storage, but attention still retrieves previous latent entries. It is not constant-size recurrent memory. Fused projections and kernels determine achieved latency. DeepSeek-V2: full MLA formulas ↗

Why RoPE complicates the identity

Rotary positional encoding inserts position-dependent rotations into queries and keys. The score contains a relative-position matrix between W_q and W_k^T. It can no longer be absorbed into one fixed query projection.

A decoupled design splits content and rotary coordinates. The unrotated content path uses the latent factorization. A smaller positional key remains separately cached. Its storage is therefore roughly r + d_RoPE elements per position, not just r. The actual layout can include additional normalization or quantization state.

GLM-5’s existing report note records a 512-dimensional KV latent and a shared 64-dimensional rotary key. Each query head has 192 unrotated and 64 rotary coordinates. A configuration’s “64 KV heads” does not by itself establish ordinary MHA storage.

DeepSeek-V2 ↗ · MLA analysis, part 1 ↗ · MLA analysis, part 2 ↗ · Positional encoding →

Local, global and cross-attention

The previous designs change stored representations. Connectivity is a separate choice: which positions can a query access? A causal sliding window of width w allows positions from max(0,t−w+1) through t. With fixed w, the attention work grows roughly as nw rather than n².

Stacked local layers expand the receptive field, but a distant token may require many propagation steps. Periodic full-context layers create direct long-range access. “Global” can mean a full attention layer or a designated token connected to every position, as in Longformer-style patterns. These are not the same layout.

MAI-Thinking-1 uses five local layers for each global layer. Its local window is 512 with RoPE base 10,000; global layers use no positional encoding. The recorded configuration has eight KV heads, head dimension 128, and RMSNorm on queries and keys. This is a tested recipe, not evidence that global RoPE always hurts.

Cross-attention changes the source

Self-attention derives Q, K and V from the same sequence. Cross-attention derives queries from one sequence and keys/values from another. The score matrix has n_q rows and n_k columns. Translation, retrieval and multimodal models use this pattern. Different modalities are not required.

Longformer ↗ · MAI recipe → · Gemma 4 recipe →

Linear attention: a different memory

For an unmasked operation without softmax, associativity gives (QK^T)V = Q(K^TV). Computing K^TV first avoids the n × n intermediate. This rearrangement does not hold for ordinary softmax because its nonlinear normalization sits between the products.

Kernelized linear attention uses feature maps φ(q), φ(k). Some maps approximate the exponential softmax kernel; others define a different operator. In causal form, the model accumulates a matrix S and a normalizer vector z.

Here φ(k) and φ(q) are column feature vectors. The state is a matrix, not a list of original token values. For fixed feature and value dimensions, state size does not grow with context length. Computation instead scales with those feature dimensions. A large state can still be expensive; “linear in length” is not a claim that it wins at every sequence length.

The cost reduction changes the memory representation. All history must fit into a fixed-size state, so exact retrieval of individual old tokens can become difficult. Copying, induction-like retrieval and long code references need targeted evaluations, not just average language-model loss.

A global low-rank argument alone is insufficient: a causal mask can make a triangular attention matrix full rank. The important restriction is the recurrent state’s representational capacity and update rule. A short causal convolution captures local order and patterns before the recurrent update. It does not limit the state to that local window.

Transformers are RNNs, 2020 ↗ · Linear-attention derivation ↗

What linear complexity holds fixed

The reassociation (QK^T)V = Q(K^TV) changes multiplication order only for the unnormalized unmasked operator. With key width d_k and value width d_v, the two large contractions cost O(nd_kd_v) rather than O(n²(d_k+d_v)). Feature construction has additional cost.

A causal kernel recurrence with r features stores rd_v matrix entries plus r normalization entries per head. Each step costs O(rd_v) for update and read. The phrase “linear in sequence length” holds r and d_v fixed. Increasing feature count to improve approximation can change practical cost substantially.

Keep the denominator when compressing history

Replacing softmax requires a declared similarity kernel. A finite feature map can define a new kernel; it does not automatically reproduce the exponential kernel. Put the standard scale inside the kernel when approximating softmax.

Initialize S and z to zero. Add the current key–value write before reading when the causal mask includes the current token. For strict past-only attention, read before writing. Reset state at independent sequence boundaries. A zero denominator needs an explicit skip, rejection or fallback policy.

The denominator preserves normalized reads as the prefix grows. Omitting z produces the unnormalized operator and changes its behavior. Fixed inference state also does not imply constant total training memory: a naive backward pass can retain every intermediate state.

Linear Transformers: kernel reassociation and causal state ↗. Feature construction, denominator policy and the backward implementation belong to the complete method.

The short convolution provides local order-sensitive mixing before compressed memory. It can handle nearby patterns without requiring the global state to encode every local relation. It does not restore exact retrieval from all historical tokens or restrict the recurrent state to only w positions.

For a fixed set of allowed keys, one softmax weight approaches one as its score gap over the others grows. A finite feature map changes this concentration behavior. The resulting retrieval limits depend on the chosen kernel and update rule, not only on matrix rank.

Evaluate copying, induction, code dependencies and long-context retrieval separately. A compressed state can perform well on average loss yet miss a particular old token. Compare complete architectures at matched training and inference conditions rather than treating all recurrent operators as one quality class.

Delta updates and bounded decay

Simple additive memory keeps writing new key–value associations, even when they conflict with existing content. A delta update first predicts the value already stored for a key. It writes the prediction error instead of the full value.

For a unit-length key, $\beta=1$ corrects that key’s current prediction exactly in this isolated ungated update. Other correlated keys can still interfere. L2-normalized keys control update magnitude; normalized queries control read magnitude. Normalization alone does not guarantee long-run stability.

Gated variants also decay parts of the state. The exact order of decay, prediction and write depends on the recurrence. Do not interchange these operations when reproducing a specific kernel.

Why a decay bound affects the kernel

Very small decay factors can create large reciprocal rescaling values in chunked algorithms. K3 bounds each step’s log decay to make more tiles suitable for dense Tensor Core operations. A per-step bound does not bound the reciprocal of an arbitrarily long product; chunk size and accumulation remain important.

A bound closer to zero restricts rapid forgetting. A much more negative bound recreates numerical difficulties. This is a model–kernel co-design choice, not only an architectural hyperparameter.

K3 combines 69 KDA layers with 24 global Gated MLA layers. Its three-to-one pattern plus a final global layer separates economical recurrent accumulation from sharp token retrieval. Its NoPE global layers do not require RoPE interpolation. This does not by itself guarantee length extrapolation.

GLM-5.3-Flash uses 34 KDA and 11 sparse MLA blocks. Its KDA blocks include width-four causal depthwise convolution, 64 heads of width 128, L2-normalized Q/K, and gated normalized outputs. Its global MLA blocks have KV rank 512 and zero rotary dimensions.

K3 report details → · GLM-5.3 implementation details →

Qwen: decay, predict the stored value, then correct it

A fixed-size recurrent memory avoids a growing KV cache, but it cannot preserve every original token independently. Qwen3.8-Flash-Next uses three Gated DeltaNet (GDN) layers per retrieval layer. The retrieval layers use full attention initially, then Qwen Sparse Attention (QSA) after continued pretraining. This is a hybrid, not a claim that recurrent memory replaces exact retrieval.

For a scalar illustration, let the old state be 2, the key be 1, decay be 0.5, value be 3 and write strength be 0.25. Decay gives 1; prediction error is 2; the new state is 1.5. Predicting before decay would give a different result. Correlated keys can still interfere in a matrix state.

The output is RMS-normalized, multiplied by a sigmoid output gate, and projected. Qwen changes the original GDN SiLU output gate to a bounded sigmoid. The model uses zero-centered RMSNorm throughout. These are implementation choices with reported gains, not interchangeable activation names.

The hybrid ablation uses 28-layer 25B-A3B MoEs: 400B tokens at 4K, then 80B at 32K. Both hybrids retain one full-attention layer in four; the sliding-window alternative uses a window of 128. Across nine selected benchmarks, mean scores are 49.87 for full attention, 51.15 for the SWA hybrid and 53.81 for GDN. GDN wins eight comparisons against full attention and seven against SWA. The comparison supports this tested hybrid; it does not identify the cause of every individual score change.

RoPE and NoPE look similar during pretraining here. After post-training, NoPE produces substantially more endless generation, so Qwen retains RoPE in retrieval layers. This does not contradict successful NoPE recipes in other models. Positional encoding choices → · Late-stage checks →.

Qwen3.8-Next architecture report, §2.1.1, pp. 3–5 (26 August 2026) ↗.

QSA: select compressed blocks, attend to original tokens

Sparse attention needs a selector. A token-level selector can itself become expensive at long context. QSA reduces that cost by compressing only the keys used for selection. Core attention still reads original token representations from selected blocks.

1. Project a query for each token and keys for the indexer.
2. Average each complete group of four keys, then RMS-normalize the pooled key.
3. Apply partial RoPE after pooling. A block uses its starting position; a query keeps its token position.
4. Sum ReLU query–key similarities across four indexer query heads, with one shared key head.
5. Select up to 512 complete blocks. Expand them into at most 2,048 original positions, then include the unfinished current block.
6. Compute sparse core attention over those original tokens.

Pooling before rotation avoids averaging vectors with different rotary phases. The final incomplete block has zero to three tokens, so the 2,048-token selection budget is not a strict bound on the complete core-attention set. At short prefixes fewer complete blocks exist. Any implementation must mask future tokens and handle an empty set of complete blocks.

Teach the selector, then adapt the backbone

QSA is introduced at 256K continued-pretraining context, not used from the first token of the main run. The teacher sums dense softmax distributions across attention heads and L1-normalizes. It then max-pools token probabilities within each block and normalizes again. Key vectors use average pooling; teacher probabilities use max pooling. These are deliberately different operations.

During sparse adaptation, compute the KL loss only over selected complete blocks. Renormalize the teacher on that selected support before comparing distributions. The indexer does not receive the dense objective over every unselected block in this stage. The report fuses sparse attention and the KL loss without materializing all intermediate tensors.

Directly using the initialized sparse selector loses RULER quality; joint adaptation recovers it. The final reported LM-loss gap is on the order of 10⁻⁴. In Table 2, short-task mean accuracy rises from 75.9 to 76.8; seven of eight scores improve, while MMMLU falls from 81.8 to 81.1. This is a measured result, not a proof that sparsity always improves quality.

RULER remains strong at 1M while 8-needle MRCR is only 26.44. A supported context length does not mean reliable retrieval on every task. Compression-factor ablations at 35B-A3B favor four-token blocks; larger blocks lose fine detail. Four indexer heads balance quality and speed. Sharing selectors across global layers performs worse at comparable indexer savings in this hybrid, where three recurrent layers separate adjacent global layers.

With fixed compression ratio r, indexer prefill remains quadratic: $O(n^2/r)$. Sparse core attention is approximately $O(nK)$ at fixed K. Compression reduces the selector constant; it does not make the whole algorithm asymptotically linear. Kernel measurements and systems → · Reuse selection during MTP →.

Qwen3.8-Next architecture report, §2.1.2, Tables 2–4, Figures 3–6, pp. 5–9 (26 August 2026) ↗.

FlashAttention and online softmax

Linear attention changes the operator. FlashAttention keeps the dense softmax rule but changes where intermediate results live. A conventional implementation writes scores and probabilities to high-bandwidth memory (HBM). Those n × n arrays create expensive traffic.

FlashAttention tiles Q, K and V into smaller on-chip memory. It maintains compact row statistics and an output accumulator rather than storing the complete probability matrix. Dense arithmetic is still quadratic; avoiding intermediate traffic can nevertheless improve runtime substantially.

Combine blocks without losing normalization

For a score block b, keep its maximum m_b, exponential sum ℓ_b, and weighted value sum u_b. When a later block has a larger maximum, rescale the existing accumulators before combining them.

The final result equals the direct softmax operation in real arithmetic. Floating-point ordering can cause small differences. A low-precision approximation such as an FP8 path introduces additional numerical considerations; “exact attention” does not mean bitwise equality for every kernel.

FlashAttention-4: when faster matmul exposes another bottleneck

The 2026 FlashAttention-4 design targets Blackwell’s uneven resource scaling. Tensor Core throughput grows faster than exponential throughput and shared-memory bandwidth. Faster matrix multiplication therefore exposes softmax in forward and memory movement in backward.

The forward pipeline alternates query tiles and separates correction work. It combines hardware exponentials with polynomial evaluation on FMA units. Conditional rescaling reduces repeated vector work while preserving the normalization bookkeeping. Approximate exponentials still require numerical validation.

Backward uses tensor memory and cooperating two-CTA matrix operations to reduce shared-memory traffic and atomic reductions. Causal and variable-length workloads also need balanced tile scheduling. The implementation uses CuTe-DSL. These changes target specific hardware resources, not a new attention distribution. FlashAttention-4 design ↗

Hardware support and kernel interfaces change. Select a kernel for the actual GPU, head width, mask, dtype and gradient requirements. Do not interpret the version number as a universal performance ranking. The next sections give a worked accumulator and the hardware/API contracts.

Watch the normalization reference change

The accumulator is easiest to inspect when the reference maximum changes. Use four scores [0,3,−1,4] and values [1,0], [0,1], [1,1], [−1,0]. Each tile contains one key. This is a numerical demonstration, not a GPU schedule or performance trace.

Tile2 changes the maximum from0 to3. Multiplying the old sums by exp(−3) expresses both tiles relative to the new maximum. Without that rescaling, the algorithm incorrectly treats the two scores as equally weighted.

Every stopped state remains readable. After all four keys, u/ℓ equals direct softmax followed by value aggregation. Reordering score/value pairs preserves the result in real arithmetic; this control changes only score order while values stay fixed, so its final answer can differ.

Hardware paths and kernel contracts

After avoiding quadratic intermediates, kernel performance still depends on how threads share work. FlashAttention-2 reduces non-matmul overhead and improves block/warp partitioning. More parallel work is useful only when register and shared-memory demand allow enough resident execution.

FlashAttention-3 uses Hopper's Warpgroup Matrix Multiply-Accumulate (WGMMA) and Tensor Memory Accelerator (TMA). Producer warps move tiles while consumer warps compute. Pingpong scheduling overlaps one warpgroup's softmax with another's matrix multiplication; further overlap within a warpgroup costs registers.

Its FP8 path uses a random-sign Hadamard transform to distribute activation outliers before quantization. An orthogonal transform applied consistently to Q and K preserves their dot products before rounding. Low-precision rounding still introduces error. These mechanisms do not imply an end-to-end speedup for every workload. FA3 design and scoped benchmarks ↗

Installation facts below were checked against the official repository on6 October2026. They are implementation requirements, not permanent properties of the algorithm.

# Official FA4 interface; requires a supported installed GPU stack.
# q: [batch, nq, query_heads, head_dim]
# k, v: [batch, nk, kv_heads, head_dim]
from flash_attn.cute import flash_attn_func
out = flash_attn_func(q, k, v, causal=True)
# Interface illustration, not executed by this website.

Check the selected API's causal alignment when query and key lengths differ. Confirm packed-sequence boundaries, head grouping, dropout, backward determinism and cache updates. Passing fewer KV heads requires compatible grouping; do not silently expand the cache.

The official repository also documents ROCm paths; NVIDIA requirements above do not describe every backend. Pin the package and benchmark the actual supported path. Official installation and API reference ↗

A framework call can choose a different kernel or fall back. Inspect the trace in Profiling and Kernel Optimization. Compare against a higher-precision reference with the same mask and scale. Measure prefill, decode, memory and end-to-end latency separately. Kernel improvements cannot remove an unrelated bottleneck.

Source builds and packaging can fail outside supported GPU/software stacks. During single-token decode, reading the growing KV cache can dominate even when score intermediates are no longer stored. Memory and compute limits → · Low-precision attention errors → · Distributed context ownership →

Sparse retrieval and index sharing

A sparse attention layer retrieves only selected context positions. This reduces the main value aggregation, but selecting positions can remain expensive. Separate the indexer cost from the main attention cost.

GLM-5’s DSA selects 2,048 positions. Its lightweight indexer has 32 heads of dimension 128, ReLU scores, a learned head mixture and causal top-k. Main attention scales with the selected count; the indexer still scans the prefix. Sparse selection can omit relevant keys, so it is not mathematically equal to dense attention.

Share decisions or compress candidates?

GLM-5.2 IndexShare reuses one selected-index list across a four-layer group. Each layer still computes its own queries, attention weights, values and outputs. The model was continued-trained with sharing; it is not an inference-only switch for an unchanged checkpoint.

GLM-5.3 IndexPool instead pools four neighboring indexer keys before scoring. Per-dimension softmax gates produce each pooled key. Selected pools expand back into original token indices, so main attention still reads raw values.

With four-token pools and a target of 2,048 raw positions, IndexPool selects up to 512 pools. The incomplete current pool contributes up to three extra visible positions. Main attention can therefore see up to 2,051 indices. A fourfold reduction in indexer candidates does not give fourfold main-attention savings.

Share memory without sharing every computation

DeepSeek-V4.1-Flash’s CSA2 distinguishes Full, Reindex and Reuse modes. Full constructs global memory and selects entries. Reindex keeps memory but recomputes selection. Reuse keeps both. Every block retains its own local window and main queries.

The first decoder scan selects 512 entries and a candidate pool of up to 2,048 eight-position blocks. Later reindexers search that 16,384-position pool. A relevant position excluded from the pool cannot be recovered later. The first scan still grows with context length.

GLM-5 and IndexShare → · IndexPool → · CSA2 and causal encoder–decoder →

Implementation and diagnostics

Start from the failure you need to solve. A cache-capacity failure, HBM-traffic failure and retrieval-quality failure require different interventions. A model can combine several methods, but each combination needs compatible masks, layouts and kernels.

1. Shapes. Query heads, KV groups, key/value widths and latent dimensions.
2. Visibility. Causal and padding masks, packed sequences, cache offsets and empty rows.
3. Numerics. Stable exponentials, normalization denominators, accumulation dtype and recurrence ordering.
4. Quality. Language-model loss plus retrieval, copying, code and context-length evaluations.
5. Performance. Measure prefill and decode separately at actual batch sizes, lengths and hardware.
6. Reproducibility. Top-k ties, deterministic paths, checkpointed state and train/inference agreement.

Do not compare FLOPs with throughput as if they were the same quantity. Sparse indexers, small matrix batches, memory movement and scheduling can dominate after the headline arithmetic decreases. Roofline reasoning →

Chronology: parallel research branches

These milestones explain distinct bottlenecks, not one sequence of replacements. Linear attention predates GQA and MLA. FlashAttention can execute a compatible head-sharing design without changing that design’s operator.

Sources and connected notes

Primary method links appear beside their explanations. Connected recipe notes give contemporary configurations and their report-specific conditions.

Source reports: Transformer ↗ · MAI-Thinking-1 ↗ · Kimi K3 ↗.

Normalization and retrieval: Entropy-invariance derivation ↗ · Softmax and length generalization ↗ · Softmax alternatives ↗ · Attention concentration ↗.

Latent representations: Cache and quality: MHA to MLA ↗ · A second rotary-encoding construction ↗.

Recurrent attention: From Performer to linear attention ↗ · Infinite-dimensional linear attention ↗ · Linear-attention history ↗ · Why short convolution? ↗ · Why DeltaNet normalization? ↗.