Positional Encodings

Where order enters attention

Unmasked self-attention without positional signals is permutation-equivariant. Reordering input tokens reorders output tokens in the same way. The operation cannot distinguish arrangements through an intrinsic index.

A causal decoder is different from an unmasked encoder. Position i sees only its prefix. Different prefix lengths create a graph asymmetry that later layers may exploit. Explicit encoding supplies a direct bias rather than requiring the network to infer every positional relation.

Sinusoidal features: several phase scales

One sine value is ambiguous: sin(30°)=sin(150°)=.5. The paired cosine values differ, .866 versus −.866. Together they locate a point on a circle. Sine-first and cosine-first layouts are conventions; use one consistently.

Every pair has a period $\frac{2\pi}{\omega}$. Faster pairs resolve fine changes. Slower pairs change over longer spans. Many distinct phases give more useful scales than repeated copies of one clock. Repeated frequencies are permitted, but reduce frequency diversity.

The model need not reconstruct an integer position from these features. It can exploit their structured changes directly. Periodic pairs and finite precision do not provide a guaranteed unique integer code over arbitrary lengths.

Why logarithmic spacing and the base matter

The normalized exponent $\frac{2i}{d}$ distributes pair indices over a logarithmic frequency range. At d=8 and b=10,000, frequencies are [1,.1,.01,.001]. Their denominators are [1,10,100,1000]. Without normalization, exponents 2i would make almost every pair nearly static.

A larger base lowers frequencies with $i>0$. It does not change the first frequency, $\omega_0=1$. Very slow pairs may contribute little variation over short training spans. The base is a spectrum parameter, not a fundamental constant or a maximum context length.

At large width with b=10,000, the slowest frequency approaches 10⁻⁴. The exact last frequency is $b^{-1+\frac{2}{d}}$, not exactly $\frac1b$. Log spacing covers several orders of magnitude; it does not prove that the model uses every pair effectively.

RoPE rotates learned content

Additive sinusoidal encoding constructs a vector and adds it to a token representation. Rotary Position Embedding instead rotates pairs in the projected query and key. Standard RoPE leaves values unrotated.

The learned content vector is not replaced with a position vector. Its orientation changes. The rotary spectrum uses the head's rotary width, which may differ from the embedding width used by additive PE.

Absolute rotations, relative interaction

Absolute positions m and n cancel into a relative displacement in the positional operator. The entire score is not a function only of distance: q and k still carry content, including contextual information from earlier layers.

Hold unrotated q and k fixed. Shifting both positions by c preserves this pair's score. Shifting only the key changes the relative angle. This local algebraic identity does not prove that reprocessing a full model at a new context produces unchanged hidden states.

Pair layouts, precision and cache offsets

Use two separate operations: construct phase tables, then rotate adjacent feature pairs. The following callable reference uses PyTorch, not a fused kernel.

import math
import torch

def rope_frequencies(sequence_length, head_dim, base=10_000.0,
                     device=None, offset=0):
    if head_dim < 2 or head_dim % 2 or sequence_length < 0:
        raise ValueError("nonnegative length and positive even width required")
    if not math.isfinite(base) or base <= 1:
        raise ValueError("finite base greater than one required")
    pairs = torch.arange(0, head_dim, 2, device=device, dtype=torch.float32)
    positions = torch.arange(sequence_length, device=device, dtype=torch.float32) + offset
    angles = positions[:, None] * (base ** (-pairs / head_dim))[None, :]
    return angles.cos(), angles.sin()

The tables have shape [sequence, head_dim/2]. Supply integer sequence length, rotary width and offset. Here head_dim means the even rotary width, not necessarily the full head width. Offset is the absolute index of the first supplied position.

def apply_rope(x, cos, sin):
    if x.shape[-1] % 2 or cos.shape != sin.shape:
        raise ValueError("even width and matching phase tables required")
    if cos.shape[-1] != x.shape[-1] // 2:
        raise ValueError("one phase per feature pair required")
    even, odd = x.float()[..., 0::2], x.float()[..., 1::2]
    c, s = cos.float(), sin.float()
    pairs = torch.stack((even*c - odd*s, even*s + odd*c), dim=-1)
    return pairs.flatten(start_dim=-2).to(x.dtype)

Use floating-point Q and K shaped [batch, heads, sequence, rotary_width]. Add two leading singleton dimensions to the tables. GQA permits different query and key head counts. Keep the same batch, sequence positions, device and rotary width; the phase tables broadcast across heads.

cos, sin = rope_frequencies(q.shape[-2], q.shape[-1], device=q.device, offset=offset)
cos, sin = cos[None, None], sin[None, None]
q = apply_rope(q, cos, sin)
k = apply_rope(k, cos, sin)

Concatenate any unrotated subspace separately. This reference is CPU-checked with float32, float16 and bfloat16 tensors. It is not a fused-kernel performance example or a GPU validation.

A split-half implementation pairs the first and second halves instead of adjacent dimensions. The weight layout, checkpoint convention and frequency tables must agree. Changing pairing alone does not preserve a trained checkpoint.

Compute phases with adequate precision before casting. Long positions can lose phase resolution in low-precision multiplication. Cache offsets must use the actual absolute position of each new key. Repeatedly rotating already-rotated cached keys creates an error.

Changing the base or scaling rule mid-sequence makes new queries inconsistent with stored rotated keys. Invalidate/rebuild the affected cache or use an implementation that stores enough unrotated information to recompute it. A fused kernel may apply rotations internally; test that path too.

Extending context changes a distribution

RoPE can compute an angle for any index. This mathematical ability is not a learned ability to retrieve or reason at that index. Longer contexts expose new relative phases, more competing keys, unfamiliar data and different numerical regimes.

Interpolation changes phase geometry. Continued training adapts the weights. Attention span and KV design determine cost. Evaluate these axes separately.

The Llama 3 report uses b=500,000, compared with the older 10,000 convention. Its 128k capability also uses staged long-context pretraining from 8k, not only that base change. A numeric base is not a certificate of a supported context window.

NoPE removes rotary phase extrapolation, but not attention dilution or unfamiliar long-document reasoning. A needle retrieval score is useful evidence for retrieval, not a complete measure of context use.

Interpolation, base changes and nonuniform scaling

Uniform interpolation maps positions from an extended interval into the old interval. It also scales a one-token gap to 1/s. Lower-frequency interpolation with higher frequencies retained can reduce this local distortion. These are tradeoffs, not universal winners.

In an integer numeral analogy, d/2 digits would span $\beta^{d/2}-1=b-1$. Rotary pairs are continuous phases, not extracted digits. This expression is not the maximum uniquely represented token index. The rough heuristic b≳L must not become a context guarantee.

High frequencies resemble low-order digits: they resolve small changes. Lower frequencies resemble high-order digits: they vary over longer spans. Uniform interpolation stretches every scale; changing the base leaves the first pair unchanged and stretches later pairs differently.

The Smol playbook's recorded ABF stages use about 2M for 4k→32k and 5M for 32k→64k. About 10M improved some long-context scores while hurting some short tasks. Its YaRN extension from 64k to 128k was more effective than to 256k. These are recipe-specific observations, not default values for every model.

Use adaptation data when the desired context is far outside training. Validate short-context recovery as well as long-context retrieval. Scaling only a frequency table does not reduce the memory or compute of full attention.

Preserve local distance, compress distant distance

With window w=4 and factor s=2, distance 3 stays 3, while distance 10 becomes 7. A negative displacement keeps its sign. The aim is clear: retain local geometry while coarsening long distances.

A nonlinear relative warp generally cannot be written as independently rotating every token at one linear absolute position. It need not satisfy f(a+b)=f(a)+f(b). Do not replace standard cached RoPE tables with this formula and assume the same efficient kernel remains valid.

The original ReRoPE series explores rectified relative treatment. Check the actual method, mask and attention implementation before reproducing it. The simple piecewise expression explains a design goal; it is not an implementation specification.

Partial RoPE is a subspace choice

Partial RoPE mixes position-sensitive and unrotated content subspaces within one head. MLA uses a decoupled rotary subspace for compatibility with latent KV caching. Neither arrangement is a fully NoPE layer.

Gemma 4's recorded recipe uses p=.25 and base 1M in global layers, with full RoPE and base 10k locally. Its reported 37.5% global-cache saving also uses key-as-value attention. Merely skipping rotations on some dimensions does not shrink a conventional full K+V cache.

Attention → covers MLA and key-as-value storage. Keep their memory reductions separate from a positional subspace's purpose.

NoPE is not position blindness

NoPE omits added position vectors, Q/K rotations, relative bias tables and distance slopes. It does not change the $\dfrac{QK^\top}{\sqrt{d_h}}$ attention operator. A causal mask still distinguishes visible prefixes.

These results do not prove pure NoPE wins at every scale. A bidirectional encoder lacks the same prefix asymmetry. Padding and packed-document boundaries change cues. Learned implicit position consumes capacity and depends on optimization.

Full NoPE retains O(L²) attention and the same projection-sized KV cache. Removing RoPE from pretrained weights is an architectural intervention, not a harmless decoding switch. Continue training or validate the intervention rigorously.

Local order and global retrieval

RoPE sliding-window layers can specialize in recent order-sensitive mixing. Global NoPE layers can retrieve by content without directly rotating similarity by relative distance. The useful division depends on span, data and training, not the names alone.

RNoPE-SWA repeats three local RoPE layers and one global NoPE layer, with a recorded local window of 4,096. The final 8B comparison uses 5T pretraining tokens and staged 8k→32k→128k extension. The earlier three-way RoPE/NoPE/QK-Norm observation experiment uses 750B tokens; do not merge their budgets.

MAI-Thinking-1 records five local RoPE layers then one global NoPE layer. Kimi K3's no-embedding design also relies on KDA recurrent decay for recency. NoPE with a recurrent state is not identical to NoPE inside a window/full-attention stack.

Qiao et al. study efficient/full-attention hybrids and report that full-attention layers carry much long-range retrieval. Their experiments favor NoPE in those layers with limited short-context degradation. This supports an ablation, not a universal placement rule.

is_global = (layer_index + 1) % 4 == 0
use_rope = not is_global
window = None if is_global else 4096
# Retain causal and document masks in BOTH layer types.

Change the positional operation, not the attention contract

This evaluation reference calls the two RoPE functions above. Use equal query/key/value head counts and compatible floating-point tensors. GQA needs a separately validated head-grouping path.

def positional_attention(q, k, v, *, use_rope=True,
                         cos=None, sin=None, attn_mask=None):
    if attn_mask is None and q.shape[-2] != k.shape[-2]:
        raise ValueError("cached decode requires an explicit position mask")
    if use_rope:
        if q.shape[-2] != k.shape[-2]:
            raise ValueError("this reference needs matching rotary positions")
        if cos is None or sin is None:
            raise ValueError("RoPE requires phase tables")
        q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
    return torch.nn.functional.scaled_dot_product_attention(
        q, k, v, attn_mask=attn_mask, dropout_p=0.0,
        is_causal=attn_mask is None)

For square uncached self-attention, the default supplies a causal triangle. An explicit mask must include every required causal, document and local-window restriction. In PyTorch SDPA, a boolean True permits attention. A supplied mask replaces the default triangle here.

Cached queries and keys can have different lengths and absolute offsets. Build their visibility mask from those positions; an upper-left triangle is not a general decode mask. The NoPE branch supports this with an explicit mask. The rotary branch assumes matching Q/K positions; general cached RoPE needs separate query/key phase arguments. The reference disables dropout for evaluation; training must select its own dropout probability.

These mask and dropout rules follow the PyTorch SDPA contract ↗. The reference is CPU-checked against explicit masked attention, not validated as a fused GPU path.

When Q/K enter a NoPE layer, they must actually be unrotated. Keep correct cache-causal masking even when no rotary offsets are needed. Fused code must not silently reintroduce RoPE.

A composition law unifies the mechanisms

Now ask which linear position transformations behave consistently under repeated shifts. Let F(s) transform a query and G(t) a key. Their score contains F(s)ᵀG(t). Require the positional operator to depend only on displacement.

Composition says that two small shifts produce the same map as their sum. X is the generator: a fixed rule for an infinitesimal change. The matrix exponential accumulates that rule over a finite displacement.

For integer positions alone, A(n)=A(1)ⁿ. A discrete invertible transition need not possess a real generator that extends it continuously. Real timestamps and learned effective time require the stronger continuous assumption.

Use one sign convention. Above $\Delta=t-s$ is key minus query. For causal decay below, age $a=s-t\ge0$ is query minus key. Replacing Δ with a negates the relevant generator; confusing them turns decay into growth.

A Toeplitz-like operator table has the same A along each relative diagonal. Actual content logits need not be Toeplitz, because q and k vary with the tokens. A causal mask removes future-key entries above the diagonal without changing the algebraic group.

Read the generator, not only the name

A real scalar block gives exponential decay or growth. A complex conjugate eigenvalue pair $u\pm\mathrm{i}\omega$ gives $e^{ua}$ times a rotation after a fixed basis change. A nontrivial Jordan block adds a polynomial factor.

RoPE chooses imaginary frequencies. Partial RoPE combines rotation blocks with zero blocks. Retention can combine rotation and decay. Data-dependent gates can define effective elapsed time, but need not form one fixed content-independent generator family.

One head can preserve content in one subspace, rotate another, and decay a third. This is a design possibility, not a reported winning architecture. Pair or scalar formulas keep the operation linear in feature width.

Additive bias through augmented coordinates

ALiBi directly subtracts a per-head slope times causal age from a logit. That additive content-independent term is not a same-width linear transformation of arbitrary content Q/K.

The construction connects additive linear bias to a unipotent group action. It is an explanation, not a recommendation to replace a cheap bias addition with dense matrix operations. Arbitrary relative-bias tables are not all generated by this construction.

The Jane Street article explains this restricted classification and its ALiBi addendum. GRAPE develops related group actions and learned subspaces beyond fixed RoPE coordinate pairs. The framework is useful for design; it does not establish a universal winning positional family.

Algebraic validity is not numerical stability

An invertible group can be stable in one direction and unstable in the other. A decaying age transform grows at negative age. Causal masking excludes future keys; bidirectional use requires a different stability argument.

A poorly conditioned change of basis can amplify norms or numerical error even when the spectrum appears benign. Pure imaginary eigenvalues alone do not guarantee orthogonal, unit-norm motion. Non-normal decaying systems can also show transient growth.

Parameterize small known blocks when possible. Direct rotation/decay/polynomial formulas can cost O(d); an arbitrary learned dense generator needs a more expensive exponential and a stability constraint.

The classification assumes linear feature maps, relative-shift consistency, continuity and invertibility on the represented space. Learned absolute embeddings, nonlinear mechanisms, arbitrary score biases, content-dependent operators and masks fall outside some assumptions. The result classifies a useful design space, not all imaginable positional encodings.

Ablate the positional contract

1. Identify whether the method acts on embeddings, Q/K, logits or recurrent state.
2. Specify positions, signed displacement, rotary width/pair layout and cache offsets.
3. Check norm/phase behavior, composition and precision over the intended distances.
4. Keep data, initialization, optimizer, tokens and attention span fixed in comparisons.
5. Compare full RoPE, pure NoPE, alternating layers and local-RoPE/global-NoPE ratios such as 1:1, 3:1 and 5:1.
6. Evaluate inside and beyond training length; measure both short-context recovery and long-context failures.

Test multi-needle/passkey retrieval at several depths, distractor-heavy document QA, relative-order tasks and code with distant declarations. Inspect attention mass, entropy, sinks and recency by layer type. A single retrieval score cannot explain all failure modes.

Measure prefill throughput, latency and cache memory at actual serving lengths. Check document-isolated causal masks during packed training. NoPE does not excuse cross-document leakage; scaling does not excuse stale-key phases.

Parallelism → handles distributed long-context work. Attention → separates operator, span and cache compression. Positional encoding changes how positions influence these operations, not every cost of using them.

Sources and connected notes

Transformer, 2017 ↗ · RoFormer, 2021 ↗ · ALiBi, 2021 ↗ · Causal NoPE, 2022 ↗ · Length generalization, 2023 ↗ · RNoPE, 2025 ↗ · Efficient/full-attention hybrids, 2026 ↗ · Translation-group derivation ↗ · GRAPE ↗ · RetNet ↗