Recurrent and Looped Models

What is repeated, and when?

These are not stages of one inevitable replacement sequence. LSTM is a classical sequence recurrence. Ouro (2025) studies latent depth recurrence. Sparse Layers (May 2026) combines that reuse with sparse experts. DeepLoop (July 2026) studies its update scaling. Recirculation and Full-Bandwidth Transformer (August 2026) explore different state paths.

Visible chain-of-thought adds emitted tokens. Latent recurrence adds internal transitions. Neither guarantees better reasoning: more computation can repeat a mistake. See Decoding and Sampling → for generation policies, and Speculative Decoding → for accelerating a target distribution rather than changing its model.

Carry a memory through tokens

An ordinary RNN repeatedly transforms its hidden state. An LSTM adds a cell state c, with explicit retention and writing. The hidden state h exposes a gated, bounded transformation of that memory.

The forget gate selects old cell content to retain. The input gate controls the new candidate write. The output gate controls which transformed cell coordinates become visible. Sigmoid maps the three gates into [0,1]; tanh bounds the candidate and visible transformation, not the cell itself.

Initialize h and c, then apply the same parameter sets at each sequence position. Gates can be computed together from z; the output gate does not have to be calculated after the new cell. A forget value near one preserves the direct cell path. A value near zero removes it.

The additive carry path helps preserve gradients, but does not guarantee stable training. Gate derivatives and all other paths remain part of the full Jacobian. Repeated forget values below one still multiply into a shrinking direct path.

At fixed input/hidden widths, processing T tokens is O(T). Dense gate work is more precisely $O(Td(d+d_{\mathrm{input}}))$. Keeping only the latest inference state needs O(d) state; returning every output or storing backpropagation activations grows with T. LSTMs have been used for forecasting, speech, translation and sequence classification; that list is not a current performance recommendation.

import torch

def lstm_step(x, h, c, W, b):
    f, i, g, o = (torch.cat((h, x), -1) @ W.T + b).chunk(4, -1)
    c_next = f.sigmoid() * c + i.sigmoid() * g.tanh()
    h_next = o.sigmoid() * c_next.tanh()
    return h_next, c_next

def lstm_sequence(x, W, b, state=None):
    T, B, _ = x.shape
    H = W.shape[0] // 4
    h, c = (x.new_zeros(B, H), x.new_zeros(B, H)) if state is None else state
    hidden, memory = [], []
    for token in x:
        h, c = lstm_step(token, h, c, W, b)
        hidden.append(h)
        memory.append(c)
    hs = torch.stack(hidden) if T else x.new_empty(0, B, H)
    cs = torch.stack(memory) if T else x.new_empty(0, B, H)
    return hs, cs, (h, c)

See a cell retain and write

The cell can hold a magnitude larger than one, while tanh bounds the visible state before its output gate. These numbers are assigned for explanation, not learned from a sequence such as [1,2,3].

More executions, not more stored weights

A looped stack stores K blocks and executes them R times. Its effective block depth is N=KR. An untied stack of N blocks stores distinct weights at each visit. A tied stack sends changed states through the same weights.

Parameter memory can shrink relative to an equally deep untied stack. Arithmetic does not shrink merely because a matrix was visited before. Sequential depth can increase latency. The KV cache may need separate states for each loop even though the parameters are shared.

Visible chain-of-thought consumes extra sequence positions, context and generation-cache work. Latent looping can refine a state without emitting those extra tokens. It still executes repeated blocks and may store separate loop caches. A vocabulary bottleneck is avoided at those internal transitions, not proof that their computation is useful reasoning.

For an MoE, distinguish total stored expert weights, active experts per visit and repeated active operations. A 6ND training proxy must count the effective active parameter operations over all repeated visits; deduplicating reused matrices understates compute. Attention’s sequence-length work and serving overhead also fall outside that simple proxy.

Follow shared blocks through three rounds

Choose when to stop

Fixed looping allocates the same maximum depth everywhere. An adaptive gate can decide that a token has received enough processing. Ouro applies a dense projection and sigmoid to the final-layer state at each loop. Its conditional halt probability is a hazard, not the unconditional chance that this loop is the exit.

During training, Ouro executes all allowed loops and computes a loss at every possible exit. It combines these losses with the learned exit masses. Early termination is an inference decision, not a shortcut in this training objective. A penalty toward a uniform exit prior discourages all training mass from collapsing onto one loop. Such collapse can reinforce itself: a loop gets more loss weight, learns more, becomes more confident and receives still more weight.

The reported inference gate uses the first loop whose cumulative exit mass reaches a threshold q. This is deterministic quantile selection, not sampling T. A uniform prior does not mean all tokens actually stop at every loop equally often.

Why not favor shallow exits from the start? An early-biased geometric prior reduces the loss weight reaching later loops. Those loops can remain undertrained, making early exits look artificially preferable. Ouro’s four-loop, 776M-parameter prior ablation reports lower loss with a uniform prior. This is evidence for that setup, not a universal rule for adaptive computation.

Stage I trains the model and gates jointly. Stage II freezes the language model and trains only the gate. Detached per-loop losses supply a signal about marginal improvement versus extra compute. The second stage changes depth allocation, not the frozen model’s per-depth predictions. Ouro: halting and training ↗

Hazards are not exit probabilities

Uniform exit mass requires hazards $\frac14$, $\frac13$, $\frac12$ and a forced final exit—not four hazards of 1/4. A low-entropy gate may be confidently wrong. Mean exit depth is a distribution statistic, not this quantile policy’s measured average serving depth.

Keep the right state for each loop

During parallel prompt processing, each loop can produce a different key/value history. A token’s later loop should attend to a compatible loop history. Replacing this with every previous token’s individually selected exit state introduces a different dependency and can prevent the same parallel prefill schedule.

Ouro’s decode experiments compare full per-loop caches with reusing the first, last or averaged loop cache. In this four-loop ablation, “last” means the fourth-loop cache, not each token’s adaptively selected exit cache. These are approximations, not mathematically identical histories. In the checked table, full cache gives GSM8K 78.92 and MATH500 82.40. First-cache reuse gives 18.73 and 8.43; last gives 78.85 and 80.40; average gives 78.73 and 78.52.

Last/average reuse cuts the four-loop decode cache by about 4× in that setup, but MATH500 drops by 2.00/3.88 points. Do not summarize this as equal quality everywhere. The paper separately reports that sharing prefill cache hurts GSM8K by over ten points. Prompt and decode cache policies must be evaluated separately. Ouro cache ablations ↗

What Ouro establishes

The original report discusses 1.4B and 2.6B models and thinking variants, with about 7.7T total training tokens. The 2.6B branch is initialized by duplicating non-embedding layers from the smaller checkpoint after an initial 3T-token stage. The later phases also improve data quality and relax the duplicated model. This is a training recipe, not a clean parameter-only comparison.

The checked revision describes early eight-loop loss oscillations, then a reduction to four loops. The 1.4B model has 24 stored layers; the 2.6B branch has 48. The reported base 2.6B model exceeds Qwen3-Base 8B on selected reasoning benchmarks; Thinking variants are separately compared with 7B–8B reasoning models. Neither comparison establishes equal overall quality or matched compute.

Table 8 lists Qwen3 8B at 36T training tokens, Gemma3 12B at 12T, and Ouro 2.6B at 7.7T. These are different parameter and data budgets. Parameter efficiency must not be read as training-compute efficiency.

In synthetic biography experiments, more loops do not add new factual storage capacity in the same way as more independent weights. Tasks that manipulate already represented facts can benefit from two or four loops. Running beyond trained depth can help easier cases, but harder cases can peak around three or four loops and then degrade. “Keep looping until correct” is not a supported policy.

Universal Transformers and PonderNet provide related recurrence/halting ideas. LoopCD extrapolates differences between recurrent-state output logits and belongs with decoding. Parameter reuse, halting and cache reuse each need their own ablation; none is a universal smaller-model substitute.

Repeat a conditional subnetwork

A dense shared block is reused with a changed input. An MoE adds another source of variation: the router can select different experts on later visits. It does not make the dense alternative’s outputs identical.

Sparse Layers compares dense Base, dense Loop, untied MoE and LoopMoE. The main loop setting stores eight layers and executes two rounds, against sixteen untied layers. It uses MHA, RoPE, SwiGLU and Pre-RMSNorm. The MoE uses top-k token-choice routing: each token selects two of eight experts, with load-balance and router-z penalties.

The authors propose that distinct expert paths relieve the expressivity constraint of tied dense FFNs. This hypothesis differs from the measured loss ranking. A dense shared FFN also receives changed states; it is not an identical-output machine.

Across most layers, 4–14% of tokens choose identical experts across the two rounds and 25–53% choose disjoint sets; most overlap partially. Layer seven is an exception: 37% exact overlap and 3% disjoint routing. The authors speculate that this layer stabilizes representations near the loop boundary. These observations show changing conditional paths, not proof that routing causes the benchmark gains. See MoE → for the router and balancing mechanisms.

Changed routing can expose four distinct experts across two visits while selecting only two per visit. Expert count alone does not establish better reasoning.

The experiments use a 10B-token FineWeb-Edu sample, GPT-2 tokenization, AdamW, WSD scheduling and μP width transfer from 128 to 1,024. Width transfer changes validation loss by under 1% relatively in the tested setting; transferring to a changed depth is less straightforward. Sparse Layers setup ↗

Quality and savings are different measurements

At iso-FLOP budgets 5×10¹⁶, 2×10¹⁷, 5×10¹⁷ and 10¹⁸, validation-loss ordering favors MoE, then LoopMoE, Base and Loop. Base and LoopMoE fitted scaling slopes are about 0.076 and 0.077: a favorable offset is not a fundamentally different scaling exponent.

MoE has the best validation loss here but the lowest Core9 score. Training objective and downstream evaluation are not interchangeable. These are small models: tested active counts reach roughly 305M and stored MoE counts roughly 711M, not frontier deployments.

Training-free early exit inspects an intermediate normalized state through the LM head. Low token entropy is confidence, not correctness. The policy exits at the first candidate boundary whose entropy falls below a threshold τ. The threshold is swept to form a compute–quality curve; no extra exit head is trained.

Why inspect loop boundaries? Every loop ends in the same physical layers used before the final vocabulary projection. The authors hypothesize that these output-facing layers make earlier boundaries more suitable for readout. Untied intermediate layers do not share that final-layer role.

The logit lens applies the final normalization and LM head to intermediate states. It then compares each resulting distribution with the full-depth distribution:

On the tested 10¹⁸-FLOP models, agreement improves sharply at loop boundaries. The analysis counts normalized JSD below 0.5 as convergence. Agreement with the final output is not agreement with the true answer. This diagnostic supports the output-facing-layer explanation; it does not guarantee every early exit is correct.

A model with R loops has R−1 early loop boundaries. Untied sixteen-layer models can exit at fifteen intermediate layers. More candidate points alone therefore do not explain the looped advantage.

More loop boundaries improve this exit tradeoff, but the full-depth quality changes too. Theoretical FLOP savings are not measured request throughput or latency. Fixed effective depth and mostly width scaling limit extrapolation to much deeper networks.

Shared weights collect cross-use updates

Reusing parameters changes how an optimizer update affects the whole unrolled computation. In an untied stack, each use receives its own gradient. With tied parameters, all use-gradients are summed into one update, and that update is read again at every use.

The issue is not gradient correlation alone. Both write directions and output sensitivities can align. DeepLoop represents their combined effect with an alignment factor κ. Under its local assumptions, κ lies between zero and the reuse count R. Decorrelated uses can give O(1); coherent alignment can give Θ(R).

These are modeled regimes, not a direct measurement of every model’s κ. Reusing one parameter object is also different from initializing several independent copies to equal values: the latter still makes separate optimizer updates.

With $R=3$, the tied expression has nine write/read pairings: three diagonal terms and six cross-visit terms. The untied expression has only the three paired terms. Alignment decides whether the cross terms reinforce or cancel.

Control aligned update growth

DeepLoop studies a residual form with a scaled skip and selected initialization gains. For $N=KR$ effective blocks and $M=2N$ residual sublayers, use $\alpha=(2N)^p$ and $\beta=(8N)^{-p}$.

The baseline and DeepLoop use the same looped backbone, tied embeddings and input-embedding RMSNorm. Data, optimizer and seed are held fixed. The baseline uses $\alpha=\beta=1$; DeepLoop enables the scaling rule. The residual expression above has no additional inner normalization. DeepLoop equations and comparison ↗

At the DeepNorm exponent, aligned reuse leaves a factor proportional to R. The DeepLoop exponent removes that growth in this local bound.

More generally, keep K fixed and assume $\kappa=\Theta(R^\gamma),\quad\gamma\in[0,1]$. Bounded growth in this local model requires $p\ge\frac{1+\gamma}{4}$.

This is not a convergence theorem. The argument assumes visit-wise update and output-sensitivity scales of $O(\beta/\alpha)$ near initialization. Hidden constants can depend on the optimizer, width and architecture. Higher p weakens the effective branch scale and can slow useful learning. The small-reuse sweep does not identify a globally optimal exponent.

At R=3 on GPT-2 small, seven exponents from 0.30 to 0.60 were tested with up to five seeds each. The short runs compare step-2,000 validation where available; failed runs use a trailing training-loss surrogate. Values below 0.45 failed in the tested seeds; 0.45 was mixed; values at least 0.50 were reliable in this setup. Comparing only converged seeds hides the failed trials.

import math

def loop_scales(K, R, p=0.5):
    if type(K) is not int or type(R) is not int or min(K, R) < 1:
        raise ValueError("positive integer block and round counts required")
    if not math.isfinite(p) or p < 0:
        raise ValueError("finite nonnegative exponent required")
    N = K * R
    return (2 * N) ** p, (8 * N) ** (-p)

import torch

def initialize_shared_once(matrices, base_initializer, beta):
    seen = set()
    with torch.no_grad():
        for weight in matrices:
            if id(weight) in seen:
                continue
            seen.add(id(weight))
            base_initializer(weight)
            weight.mul_(beta)

def residual_sublayer(x, branch, norm, alpha):
    return norm(alpha * x + branch(x))

def run_shared_stack(x, blocks, rounds):
    if type(rounds) is not int or rounds < 1:
        raise ValueError("positive integer round count required")
    blocks = tuple(blocks)
    if not blocks:
        raise ValueError("at least one physical block required")
    for _ in range(rounds):
        for block in blocks:
            x = block(x)
    return x

Compare the depth-dependent factors

The formulas show why a reuse-sensitive scale was proposed. They cannot tell you whether a trained checkpoint will have a specific gradient magnitude, loss or stability boundary.

Evidence for residual scaling

The runs use context 1,024 and 100K updates. Small uses four H200 GPUs; medium uses eight. Non-looped results are nearly tied. The small-model advantage peaks at R=5; the medium-model advantage grows through R=7. At R=7, the medium checkpoint improves seven of eight tasks in both zero- and one-shot evaluation. PIQA is the exception. The harness uses acc, not acc_norm, and an unweighted eight-task mean. This is not a guarantee on each task, seed or model scale.

For the hierarchical reasoning model, distinguish forward unrolling from gradient-visible depth. Earlier cycles are truncated; the analyzed last segment has M_g=24 residual sublayers and N_g=12 blocks. Per-module reuse is what matters, not a single arbitrary whole-program loop count.

The original note records a four-seed check around the budget-two setting with approximately 0.5-point standard deviation. These conditions remain distinct from the single-seed language-model table. Local initialization analysis, small models and particular optimizer settings do not establish frontier/MoE convergence or a universal training recipe.

Refine a cached interpretation

A token’s shallow state is cached before its deeper interpretation is complete. Later attention can revisit the cached state, but ordinary caching does not retroactively replace it with the deeper interpretation. Think of “bank” in a fishing context: an earlier layer may retain an ambiguous representation even after later layers select the river meaning. A later ATM question can reactivate the financial association. The motivating critical-token activation-patching study reduced contextualization errors by 60%. That targeted intervention differs from applying recirculation to every token.

Recirculation sends a deeper residual at source layer s into an earlier destination d<s, then reruns downstream computation. The basic method freezes the language model. It is not merely repeating a whole shared stack before every readout.

Norm matching makes α more interpretable. It does not prove that coordinates at the two depths represent the same concepts. Layer choice matters. Reported calibrated pairs are 11→4 for Gemma3 1B, 18→9 for 4B and 35→16 for 12B; layer-zero destinations work poorly in the tested sweep.

Calibration uses 1.5M tokens from arXiv, C4 and PG-19. Thus “training-free” still includes a calibration search. The fixed setting uses $\alpha\approx0.15$; the 1B model benefits from convex $\beta=1-\alpha$, while larger tested models benefit from $\beta=1$ residual addition. For 1B, ramp α over the first ten positions using the rule below. These are model-specific choices.

The fixed mixer adds no projection between depths. Residual additions make shared feature directions plausible, but normalization and depth-specific roles can change their interpretation. The layer sweep is evidence of compatibility, not a proof that all layers share a semantic coordinate system. Recirculation mechanism ↗

Mix directions, not arbitrary magnitudes

def mix_for_recirculation(h_dst, h_src, alpha, beta):
    # Matching [..., d] tensors; norm arithmetic uses float32.
    eps = 1e-6
    dst_norm = h_dst.float().norm(dim=-1, keepdim=True)
    src_norm = h_src.float().norm(dim=-1, keepdim=True)
    scaled_src = (h_src.float() * (dst_norm / src_norm.clamp_min(eps))).to(h_src.dtype)
    return beta * h_dst + alpha * scaled_src

Readout and refinement happen at different times

Looping processes the current latent state before its selected readout. Feedback carries a completed previous state into the next position. Recirculation uses an ordinary readout and refines history for later positions. A speed comparison is meaningless until these dependencies are specified. Visible reasoning also cycles through vocabulary readout and embeddings; it spends context positions. Residual-space state maintenance and visible reasoning can coexist, rather than replacing one another.

Exact recirculation must make the relevant refined keys and values available before later attention consumes them. The paper discusses overlapping the current ordinary pass with the previous token’s refinement. Such overlap can hide latency only when hardware has capacity; it does not remove arithmetic, energy or memory traffic.

The studied method uses one extra recirculation, not unlimited repeated refinement. Exact sequential prompt processing is expensive. Proposed blockwise prompt recurrence weakens the dependencies and was not fully evaluated; a whole-prompt block is not equivalent to exact token-by-token refinement. Do not claim a working high-throughput cache engine from a vector-mixing example.

Capture the residual outputs at s and d. Mix them, rerun blocks after d, then replace affected history entries before later attention reads them. The ordinary first-pass logits remain the readout. Define this order against the model’s cache API; appending another copy of the token changes the sequence instead of refining its state.

With r extra recirculations, the schedule needs r+1 stack-like executions per input step; only r=1 is evaluated. Extra paths and iterations are proposals. Overlap consumes capacity that could serve another request, so similar single-request latency does not imply equal throughput, energy or memory.

Learn how much state to recirculate

An adaptive mixer reads concatenated source and destination states, then produces feature-wise $\boldsymbol\alpha$ and $\boldsymbol\beta$. The reported network uses LayerNorm, two hidden-width GELU stages and a final $2d$ sigmoid output, initialized near $\alpha=0.1,\quad\beta=0.9$.

The two coefficient vectors are independently bounded in [0,1]; their sum is not constrained to one. This is not necessarily a convex combination. The full-width adapter adds roughly $5d^2$ matrix weights, so it is not free merely because the base model is frozen.

import math
import torch

def make_adaptive_mixer(d):
    net = torch.nn.Sequential(
        torch.nn.LayerNorm(2*d), torch.nn.Linear(2*d, d),
        torch.nn.GELU(), torch.nn.Linear(d, d),
        torch.nn.GELU(), torch.nn.Linear(d, 2*d))
    with torch.no_grad():
        net[-1].weight.zero_()
        net[-1].bias[:d].fill_(math.log(0.1/0.9))
        net[-1].bias[d:].fill_(math.log(0.9/0.1))
    return net

def adaptive_mix(net, h_dst, h_src):
    logits = net(torch.cat([h_src, h_dst], dim=-1))
    alpha_logits, beta_logits = logits.chunk(2, dim=-1)
    alpha = alpha_logits.sigmoid()
    beta = beta_logits.sigmoid()
    return mix_for_recirculation(h_dst, h_src, alpha, beta)

Training uses 250 documents from each of three datasets, 1,024-token windows, 100 steps, batch 32 and AdamW with learning rate 3×10⁻⁴ and weight decay 10⁻⁴. Backpropagation runs through the recurrence to train the mixer. Constant scalar, conditional scalar, constant vector and conditional vector ablations separate adaptability from extra parameters.

On the reported nine-dataset mean, conditional feature vectors reduce perplexity by about 23%, compared with 8.5% for fixed mixing and 21.6% for full fine-tuning in that comparison. The perplexity adapter experiment uses Gemma3 1B PT. These are not the same aggregation as the ten-dataset fixed-mixing table. Task-specific calibration and downstream adapter training can also change transfer results.

Where recirculation helps—and where it does not

Negative reductions mean worse perplexity. The authors associate LAMBADA’s mixed result with short sequences and tokenization artifacts. That explanation is a hypothesis, not a correction to the negative results. In one 1B experiment, temperature calibration alone gives 8.48%, recirculation 14.21%, and both 19.55%. This shows the gain is not merely the same temperature adjustment, not that every model gets this gain.

Effects persist across roughly 20–200 positions in lag analyses, with tested short-context benefits through 256 positions. The first roughly ten tokens can worsen for 1B. Adjectives, adverbs and verbs benefit more than some numerals, determiners and pronouns; plural nouns benefit more than singular nouns. These are observed patterns, not a proved semantic mechanism.

On Rules, 4B instruction-tuned accuracy moves 82.3→87.3 with fixed settings and 90.8 with task tuning; 12B moves 93.0→98.4→99.4. The task-tuned numbers are not general-purpose held-out settings. Contextual questions improve on two of three tests for 1B/4B, but 12B worsens on two tests and is near ceiling on the third.

For 4B single-token benchmarks, six of eight improve modestly: MMLU 57.90→58.28, ARC-E 81.78→82.07, ARC-C 54.44→54.86 and PiQA 79.98→80.52. WinoGrande and HellaSwag worsen.

The adaptive pass@1 gain is about 21% relative accuracy but 8.8% relative error reduction. At pass@128 the error reduction is about 20.9%. Do not interchange those denominators.

Smaller ARC training can cause regressions; broader MMLU-based adapter training transfers more consistently in reported tests. Limited sweeps on Ministral3, Pythia, Qwen3, Phi2 and other Gemma families do not establish universal compatibility. The cross-family comparison reports less than about 0.5% reduction outside Gemma, versus roughly 5% for Gemma3 1B; normalization and coefficients were not extensively retuned. Gemma’s Peri-LN or data recipe is a possible explanation, not an isolated cause. A training-free layer sweep found robust recirculation regions across Gemma3 sizes, unlike inserting repeated layers into those already-trained models. This does not contradict gains from models trained for looping. The largest tested base is 12B, not a frontier MoE with production long-context serving.

Carry a wide state into the next token

Ordinary autoregressive generation exposes a sampled token to the next position. The previous top-layer state remains available indirectly through layer-specific attention history, but is not the next token’s bottom-layer input.

Full-Bandwidth Transformer adds a direct top-to-next-bottom channel. First compute hₜ₋₁ and sample token xₜ. Its embedding eₜ gates the projected previous state:

This is asymmetric: the token selects a gate, while the previous state supplies the value. An additive embedding path could let an ordinary pretrained model suppress feedback and preserve its familiar input. The proposed gate removes that direct value shortcut; it does not guarantee useful state learning. Removing that state path removes the value contribution. A sum of state and embedding is a different mechanism.

The state is already a deterministic function of the prefix: feedback adds computational access, not new external information. It can carry a wide continuous representation without emitting many explanatory tokens. A model can still need visible reasoning tokens or produce an incorrect latent update.

At $d=1536$, two matrices add about 4.72M weights. The paper’s under-1% estimated decode-FLOP overhead is not a measured end-to-end latency guarantee. The ordinary KV layout remains, with one additional latest d-vector per request. The projections add $2d^2$ parameters, but the extra buffer and projection traffic are separate serving costs. Full-Bandwidth Transformer ↗

Use feedback at the prompt boundary

SOFT puts a boundary between prompt tokens that had no feedback and subsequent tokens that do. FUSED prepares a more compatible feedback prompt history, but roughly doubles forward prefill work. It does not double every generated token’s stack executions.

The carried state belongs to the request. Reset it for new requests; reorder it with batches and beams; restore it on speculative rollback. A single global buffer shared between requests leaks state. Fixed-address CUDA-graph buffers and EAGLE/MTP integration are implementation proposals in the original, not a verified released serving engine.

The first position has no previous-token state. It needs an explicit ordinary/start-state convention. Prompt processing must not use the final state of a future position as an earlier token’s input.

Train a sequential channel in parallel passes

Exact feedback over a whole training sequence would serialize its tokens. The training approximation instead runs several parallel causal passes. Pass one uses embeddings. Later passes read the previous pass’s top state shifted by one position.

After K passes, the explicit feedback chain reaches at most K−1 previous positions. That is not the total receptive field: attention can still read the causal prefix. Training keeps gradients through earlier passes; detaching them would change the objective.

import torch

def feedback_inputs(e, h, W_u, W_g, prefix):
    previous = torch.cat([torch.zeros_like(h[:, :1]), h[:, :-1]], dim=1)
    values = previous @ W_u.T
    gates = torch.sigmoid(e @ W_g.T)
    fused = values * gates
    positions = torch.arange(e.shape[1], device=e.device)[None, :, None]
    return torch.where(positions < prefix[:, None, None], e, fused)

def feedback_objective(e, run, loss_fn, fuse, prefixes):
    h = run(e)
    first = loss_fn(h)
    later = []
    for prefix in prefixes:
        h = run(fuse(e, h, prefix))
        later.append(loss_fn(h))
    return first + torch.stack(later).mean() if later else first

The reported loss averages the feedback-pass losses with λ=1, rather than blindly summing them.

Random prefix mix-in leaves a plain prefix and applies fusion to the suffix, matching SOFT’s prompt boundary. The averaged term has weight one regardless of how many feedback passes are present. The first-pass loss preserves STANDARD as a fallback and lets the authors check ordinary decoding.

The schedule starts with ordinary pretraining, then introduces two-pass batches and a small fraction of three-pass batches. Feedback therefore need not be active from initialization. The start point and mixture are heuristic, not a derived optimal schedule.

A 75%/22%/3% mixture of one/two/three-pass batches averages 1.28 forward passes. Thus 200B unique tokens correspond to 256B forward token-pass equivalents; 400B to 512B. These are not full backward FLOPs or measured wall-clock costs. One published 10B/100%-three-pass row lists 40B rather than the arithmetically expected 30B; do not build a general accounting rule from that anomalous row.

Make repeated feedback survive its own inputs

A channel trained only once can encounter its own outputs repeatedly at inference. A two-pass 75%/25% mixture becomes unstable at large feedback depth in the reported tests. Replacing three percentage points with three-pass training improves that behavior.

The report checks 30 feedback prefill iterations and an appendix’s 1,000-iteration stress test. A plateau in those tests is not a global contraction proof or evidence for arbitrarily long agent trajectories.

Other recipe choices include O(1) depth scaling, RMSNorm around fusion, compatible tied input/output embedding coordinates and uniform jitter with $\sigma=0.02$. Equation 13 adds jitter to the fused input, before the transformer; adding noise before projection and gating is a different perturbation. These can help the tested system; tying or normalization alone does not prove stability.

The experimental model is about 1B, with 24 layers, $d=1536$, gated FFN width 6,656, 16 query heads, eight KV heads, vocabulary 100,352 and context 8,192. Local attention uses a 2,048 window with a global layer every sixth layer. Gated GQA, head gates, Q/K normalization, RoPE and tied embeddings are part of the backbone; they are not all novel consequences of feedback.

Matrix weights use NorMuon (learning rate 10⁻², decay 0.01); other weights use Adam (5×10⁻⁴, initially zero decay). WSD warms for 200 steps and cools for 25%; z-loss of 10⁻⁵ begins during cooldown. Weight decay follows the learning-rate decay under AdamC. Data uses the Phi-4 mixture. Batch size is roughly 300K tokens versus 1.2M for the 1T baseline—a comparison confound.

Separate feedback gains from extra training

At 200B unique tokens, the original report records base MATH500 pass@1 0.27→SOFT 0.37, HumanEval pass@3 0.31→FUSED 0.34 and MBPP pass@3 0.38→0.40. Code pass@3 is estimated from ten rollouts with separately tuned temperatures from {0.3,0.5,0.7}; it is not greedy accuracy.

The feedback columns select the best reported feedback mode for each task; they do not describe one fixed mode across all benchmarks. GSM8K and MATH500 use pass@1; HumanEval and MBPP use pass@3.

Selected runs extend context from 8K to 32K with 12B tokens, then use 6B instruction-tuning tokens; these stages use three passes. At fixed checkpoint, enabling feedback is a cleaner mechanism comparison than contrasting different training-token totals. The latter also changes repeated-pass training, batch size and FUSED prompt work.

Non-generative evaluations report selected 100B-feedback versus 200B-standard and 200B-feedback versus 400B-standard similarities. Most recurrent-prefill gain occurs on the first added pass, with diminishing returns afterward. This is not a universal 2× reduction in training FLOPs. No end-to-end memory/throughput/latency benchmark is supplied by the original note’s report version.

Completion tracking tests whether a completed counter equals a required counter. Delayed memory retrieves a bit after irrelevant operations. Latest-write tracking retrieves the most recent value assigned to a queried register. Linear probes at the final position test accessibility at different residual depths. An extra feedback prefill pass makes layer-zero completion 99.6% and delayed-bit retrieval 100% in the cited probes. That does not prove the LM head uses the channel optimally on general language tasks.

Shorter math completions appear in base-model SOFT tests, but that effect disappears after conventional instruction tuning. Off-policy reasoning traces are the authors’ explanation, not a demonstrated causal result. On-policy RL and feedback-specific data are proposals. T²MLR carries a previous middle-layer state to an earlier middle layer. Latent Recurrent Transformer explores cross-position injection through attention/residual paths. These are different placements, not evidence that top-to-input is globally optimal.

Version 1 studies approximately 1B models, with main recurrent runs up to 400B tokens. It supplies neither released weights nor an official end-to-end implementation. Free-form tests cover four tasks. Fusion form, connection placement, loss weights and stabilizers are not exhaustively isolated. Peak training memory, wall time and distributed serving costs remain unmeasured.

A fixed-loop model recipe

Nanbeige4.2 combines two loop passes with a compact parameter budget, rather than Ouro’s adaptive halting. The original configuration records 22 physical layers, effective depth 44, hidden width 3,072, FFN width 10,752, 48 query heads, eight KV heads, head width 128 and vocabulary 166,144.

The released implementation resolves the apparent width mismatch. Query heads occupy $48\times128=6144$ coordinates, while the residual stream stays 3,072 wide. Its explicit head-width field overrides the usual hidden-width/head-count fallback. The output projection maps the concatenated head results back to the residual width; $d=h\,d_{\mathrm{head}}$ is not a required identity.

Each KV group serves $\frac{48}{8}=6$ query heads. Wider queries do not change the number of stored KV heads. This shape check explains the architecture, not its quality or measured throughput. Pinned checkpoint configuration ↗ · Pinned attention implementation ↗.

The report lists about 3B non-embedding and 4B total parameters, 28T pretraining tokens and 256K context, with thinking and non-thinking modes. Reused layers do not turn 28T unique tokens into 56T independent data.

Two loops are reported as the chosen quality/efficiency tradeoff; additional loops bring marginal gains and instability. The quoted roughly 75% token-processing efficiency lacks the fully controlled serving setup needed for a universal speed claim. Shared loop KV saves about half the loop cache but weakens gains; the released recipe retains separate loop caches according to the original note.

From-scratch training outperforms the reported upcycling trial, but architecture, data and recipe details are not isolated. Math/code/STEM, synthetic QA and agent data change alongside the model. Its environment construction, SFT/RL stages and full benchmark comparisons are in the Nanbeige4.2 recipe →. Looping alone does not explain those training choices.

Checks before claiming a saving

First identify the state and dependency: current token across depth, previous token across time or refined history for later attention. Test a future-token perturbation: earlier predictions must not change. Test start-state initialization and request reset separately.

For tied layers, verify parameter identity and optimizer deduplication. Log per-round state RMS, branch RMS, gradient RMS and update/state ratios. Count every active matrix visit. In truncated training, record exactly which cycles receive gradients.

For halting, conserve exit mass and force the budget’s last exit. Validate thresholds against cumulative mass. Compare full and reused caches independently for prompt and decode. For recirculation, test layer indexing, norm clamps, cache invalidation and the readout/refinement boundary.

For feedback, shift previous states, preserve plain-prefix boundaries and compare one-, two- and three-pass losses. Verify carried states under batching, beam reordering and speculative rollback. A correct vector formula is not a correct distributed scheduler.

Finally measure unique tokens, token-passes, actual training FLOPs, peak memory, prefill/decode latency and throughput at matched quality. Parameter memory, KV storage and serial depth are different budgets. Link architecture to Parallelism → rather than interpreting every theoretical FLOP reduction as a deployment win.

Sources and connected topics

Ouro (2025) ↗ · Sparse Layers (2026) ↗ · DeepLoop ↗ · Recirculation ↗ · Full-Bandwidth Transformer ↗ · Microsoft Research publication ↗

Universal Transformers ↗ · PonderNet ↗ · DeepNet / DeepNorm ↗ · Nanbeige4.2 model ↗

For the broader compute budget, connect Scaling Laws to reasoning compute allocation and serving economics. Compact model recipes preserve data and post-training choices that looping alone cannot explain. Concept-space models change the representation unit instead of adding recurrent depth.

Recirculation’s direct mixing differs from Attention Residuals, which retrieves earlier depth states. Connect its coordinate assumptions to normalization and its retained path to residual connections.

Connect gates to activation functions, output probabilities to binary cross-entropy, and update monitoring to gradient diagnostics. These are different roles: an LSTM gate is not a supervised binary target.

The bounded demonstrations illustrate mechanisms. Benchmark tables contain reported results under their stated conditions, not outputs of the animations.