Frontier Model Training Recipes

What can a complete recipe tell us?

A complete recipe is a dependency chain. Data determines which behaviors the model can learn. Architecture and kernels determine feasible length and throughput. Pretraining supplies exploration support. Post-training changes behavior, and serving determines the numerical path and agent interface.

A useful report can reveal this chain even without an isolated ablation for every component. It cannot then prove that one new activation caused the entire gain. Distinguish disclosed configuration, reported observation, author explanation and a proposed adaptation.

Some sources are technical reports; GLM-5.3-Flash instead has a release post, weights, configuration and implementation. Those establish different things. A configuration can prove a layer dimension, but cannot disclose an absent training mixture.

Compare the same quantities

MAI's total is approximate. GLM-5's parameter convention includes MTP but excludes embeddings/output head in the report. Kimi and MiMo use their own reported active conventions. DeepSeek's lookup memory is separate from its backbone. These are not equal-cost models.

Training exposure is not unique corpus size. MAI's 30T excludes separately stated mid-training stages. GLM's rounded stages total 28.55T rather than the approximate 28.5T headline. Do not add MiMo Flash's training to Pro's budget or borrow Turbo's context for dense Qwen.

Stored capacity is not active compute

Trace the stages without inventing duration

Two hybrid layouts, two final-layer choices

KDA compresses the prefix into a recurrent state. Global or sparse MLA retrieves particular earlier tokens. A three-to-one cadence combines those roles; it does not prove that three-to-one is optimal elsewhere.

MAI and MiMo instead use mostly local attention with periodic global layers. DeepSeek shares global caches and selection across depth. These change different objects: memory representation, attended positions, or repeated selection work. Attention →

Llama 3: scale a dense baseline

Llama 3's 8B, 70B and 405B dense models use GQA to reduce KV-cache storage and communication. The recipe trains on about 15T tokens. For 8B, that is about 1,875 tokens per parameter, rounded to roughly 2,000 in the original discussion. Continued improvement at high exposure does not identify a universal compute optimum.

A rough 20–25-token-per-parameter training-only reference would allocate 160–200B tokens to an 8B model, far below 15T. That reference is not a stopping rule: deployment cost, data quality and the intended model size change the desired exposure. Scaling Laws →

The report describes a predominantly English corpus, but these four categories do not establish a 95% English / 5% non-English split. Multilingual filtering and task-specific evaluations also do not guarantee equal capability in every supported language. High-quality mathematical and reasoning sources receive more weight later.

The design couples dense scaling to execution and context adaptation. The next two sections show why the system and schedule are part of the model recipe. Llama 3 405B source ↗

Llama: map four parallel dimensions to the cluster

The 405B run uses 16,384 H100 GPUs and the logical grouping order [TP, CP, PP, DP]. RDMA over Converged Ethernet (RoCE) provides a 400 Gbps interconnect. This is a fabric specification, not a measured all-reduce bandwidth.

For pipeline balance, first and last stages can omit a transformer layer to account for embedding and loss work. Equal layer counts would not imply equal stage time. Parallelism → explains ownership, communication and schedules.

RoCE traffic between a GPU pair splits into sixteen subflows across equal-cost multipath (ECMP) routes. This reduces concentration on one path; it does not multiply each link's physical capacity. Tectonic, a distributed filesystem, handles checkpoint I/O. Checkpoint frequency trades save pauses against work lost after failure.

The reported 38–43% BF16 model FLOPs utilization (MFU) depends on sequence length and the parallel configuration. It is not calendar goodput, and excludes other causes of lost productive time. Utilization and goodput →

Llama: expand context, then finish the recipe

Start at 8K rather than paying quadratic attention work at 128K throughout the run. Context then grows through six stages to 128K over roughly 800B training tokens. These tokens belong to the overall exposure; they are not another complete pretraining budget.

RoPE uses base 500,000. The report lists that base in the architecture settings; do not imply it changes only at the final context-extension stage. Position extension → and Attention → explain the position and cache contracts.

The final 40M-token anneal lowers the learning rate linearly to zero, upweights high-quality sources and averages checkpoints, while retaining 128K context. Keep this final phase separate from the report's 40B-token data-quality probe experiments.

Alignment repeats rejection sampling, supervised fine-tuning and direct preference optimization. This differs from assuming that every frontier assistant uses online GRPO. Dense architecture, data selection, staged adaptation and operational accounting form one connected recipe. Llama 3 405B source ↗

Qwen2.5: data and staged alignment

Qwen2.5's open dense sizes are 0.5B, 1.5B, 3B, 7B, 14B, 32B and 72B. Turbo and Plus are separate hosted MoE variants. The dense decoder uses grouped-query attention (GQA), SwiGLU, rotary position embeddings (RoPE) and pre-RMSNorm. Byte-level byte-pair encoding uses about 151K regular tokens.

The 18T data recipe combines Qwen2-Instruct-assisted quality filtering, domain rebalancing and specialized Qwen2.5-Math and Qwen2.5-Coder data. E-commerce and social-media volume can contain repetitive templates; downsampling those domains changes the useful mixture, not just its size. Synthetic data is filtered, not automatically accepted. The report names Qwen2-72B-Instruct among synthetic generators, rather than establishing that this one checkpoint performs every quality assessment.

ABF changes the rotary frequency base; YaRN and DCA modify extrapolation and attention at inference. The 40/60 mixture belongs to Turbo's progressive stages, not a disclosed dense-model mixture. Position extension → explains the reusable mechanisms.

Supervised fine-tuning covers long responses, mathematical and logical reasoning, structured data and coding in nearly 40 programming languages. The primary report says over one million SFT examples; “40+ languages” is an overstatement of its coding scope.

1. Offline DPO: preference data targets reasoning, factuality and instruction following. The Online Merging Optimizer belongs to this offline stage, despite its name.
2. Online GRPO: group-relative policy optimization uses reward-model feedback for response quality. It removes a separate value critic, not rewards. Do not infer that this general online stage uses mathematical step rewards from a named math reward model. Its named specialized math reward model appears in synthetic-data filtering.

The sequence matters: demonstrations initialize useful behavior, offline pairs train comparisons, then fresh group samples receive online feedback. These stages are not interchangeable labels for one update. Preference Optimization → and Reinforcement Learning → give the objectives. Qwen2.5 72B source ↗

Llama 3 recipe → provides a dense comparison. Distributed Training → supplies the connected execution concepts.

MAI: build a repeatable development loop

MAI treats model building as repeated controlled improvement. Its priorities include learning capabilities in-house, keeping the stack simple and confirming changes with scaling ladders. A proxy win is promoted only after checking transfer to the intended regime.

The internal held-out NLL objective weights Coding 0.5, STEM 0.175, Math 0.175, General 0.1 and Multilingual 0.05. These evaluation weights express capability priorities; they are not the training mixture. Public generative benchmarks still matter for final behavior.

Efficiency gain compares a candidate's cost with the baseline cost needed to match its loss. EG_FLOPs and EG_Time separate model arithmetic from real implementation. A fitted floor is not known irreducible entropy. EG=1.3 means the baseline needs 30% more cost, or the candidate saves about 23.1% against that baseline.

Pretraining uses public/licensed human-generated sources, detects AI-generated content and avoids packaged open-source corpora. Extraction varies by source: structured parsers, custom extraction, agents or retained markup. Exact, fuzzy, template and semantic deduplication use a global retention priority.

Mixture search includes epoch caps and scale confirmation. A STEM-heavy proxy mixture can lose to a code-heavier one at larger scale. The disclosed final percentages are Code 54.6, STEM 15.8, Math 5.4, Web 14.9, PDFs 4.7, Books/journals 3.1 and other multilingual 1.6. Rounded percentages sum to 100.1, not exact 100. Math repeats 5.28× and code 2.22× while web/PDF are sampled below one epoch. MAI-Base-1 / MAI-Thinking-1 source ↗

Scaling Laws → and Training Data → contain the reusable models.

MAI: connect architecture to the actual run

The approximate 1T-total/35B-active, 78-layer backbone combines five local GQA layers per global NoPE layer. Local windows are 512 with RoPE base 10K; attention has eight KV heads, head width 128 and Q/K RMSNorm. Tied embeddings and bias-free blocks accompany the attention design. Dense FFNs interleave with dropless top-eight/512 latent MoE. Routing reads the original representation before shared down-projection.

The reported ladder's 100–200 TPP region is source-specific, not the universal Chinchilla ratio. Main $\frac{30\times10^{12}}{35\times10^9}\approx857$ active TPP differs from a total-parameter denominator. Production overtraining and wall-clock efficiency alter the desired design point.

Main pretraining uses 30T at 16K on 8,192 GB200 GPUs, followed by 3.4T at 64K. Final 256K extension uses 4,096 GPUs; the main table gives 150B while the selected final recipe gives 140B. Keep the scopes separate. Most extension NLL gains arrive early; representation calibration is the authors' explanation, not a theorem.

AdamW uses $\beta=(0.95,0.925)$, $\epsilon=10^{-8}$, clipping 1, peak learning rate $2\times10^{-4}$ and minimum $2\times10^{-5}$, about 12B warmup tokens, cosine decay and dropout 0.15. Less aggressive decay is reported to help later RL. Zero attention-output RMSNorm gain initially avoids near-uniform mean pooling that can collapse token diversity and MoE load.

Selective FP8 forward/input-gradient arithmetic retains BF16 weight-gradient operands and FP32 sensitive paths. YOLO uses descriptive sharding, explicit communication, offload/recomputation and full-state deterministic recovery. Goodput includes startup, replay and degraded operation rather than only kernel MFU.

RL adds adaptive entropy, an outer ratio safety bound, pass-rate filtering and top-p support replay. Self-distillation transfers later-climb behavior, formatting and gains to new base checkpoints; diverse prompts matter more than repeatedly sampling one prompt. Short old traces can forget long context. These are continuation tools, not disclosed proof that the complete recipe is independently reproducible. MAI-Base-1 / MAI-Thinking-1 source ↗

MAI: decide what an improvement means

The development loop needs a cheap signal before it can test many mixtures. MAI uses held-out negative log-likelihood (NLL): it evaluates prediction of reference tokens without generating and grading a complete answer. This reduces generation cost and sensitivity to answer formatting. It does not replace final behavioral evaluations or eliminate contamination.

A scaling ladder then tests several model sizes at a fixed tokens-per-parameter regime. Comparing curves asks whether a change survives scale. Comparing one proxy point cannot answer that question. The same architecture must also be tested at the intended training density.

At EG=1.3, the baseline needs 30% more cost than the candidate. Relative to that baseline, the candidate saves about 23.1%, not 30%. Fit FLOP and wall-clock curves separately: an arithmetic improvement can lose its advantage in an inefficient implementation. Scaling Laws →

The data loop has the same discipline. Boilerplate removal precedes exact, MinHash-LSH fuzzy, template-level and embedding-based semantic deduplication. A global drop order keeps overlapping content in the preferred source. Otherwise, a local dataset improvement can merely transfer overlap from another dataset. Larger models can exhaust duplicated content sooner; deduplication therefore changes effective novelty, not just storage.

Mixture search varies both weights and epoch caps, first locally and globally with small models, then at larger scale. The reported STEM-heavy versus code-heavy ranking inversion is a concrete reason to repeat the scale test. Keep raw markup when parsing would destroy useful structure; extraction quality depends on the source. Training Data → MAI technical report ↗

MAI: keep a long RL climb informative

The GRPO-style update uses token-level policy ratios with a response-level group-normalized advantage. The task reward supplies direction, but it does not by itself keep a long run stable. MAI adds controls for entropy, extreme ratios and uninformative groups.

These controls solve different problems. Filtering cannot repair stale probability records. Entropy control cannot make an incorrect verifier reliable. This comparison explains their roles; reproducing the climb also requires the exact objective, coefficients and sampling policy. Reinforcement Learning →

Self-distillation makes continuation possible after the climb changes the model's behavior. It moves prompted behavior into native chat formatting, recovers from unstable runs and transfers gains onto new pretraining or mid-training checkpoints. The report finds that roughly one million reasoning traces can suffice in its experiments. Later-climb traces and diverse prompts matter more than many repeats of one prompt.

The trace set also defines what survives. Distilling only older short-context traces can forget long-context behavior. Preserve representative long-context tasks when transferring a checkpoint; do not assume the architecture's context capacity guarantees retained behavior. Distillation → MAI technical report ↗

MAI: count useful training, including recovery

The YOLO stack supports custom kernels, multiple parallel dimensions, dropless MoE and activation offload/checkpointing. Its sharding annotations describe tensor layout. They do not silently insert communication into the graph. Explicit communication makes synchronization costs visible and avoids accidental barriers.

The precision state is part of recovery. Restoring weights while forgetting FP8 scale history or random-number state does not restore the same run. Determinism also requires fixed batch order, deterministic reductions, stable MoE top-k tie-breaking and a stable collective topology. These controls help test an infrastructure change, not only repeat a final score.

Asynchronous distributed checkpoints reduce pauses. Hot-standby-style recovery reduces restart delay. Deterministic comparisons check that the restart preserves computation. Failure still costs replayed work, startup time and possibly slower placement after recovery; a fast steady-state kernel hides none of these losses.

Model FLOP utilization (MFU) describes arithmetic efficiency during execution. Goodput asks how much of the full run advances the intended training job. Both matter: high MFU between failures can coexist with poor goodput. Distributed Training → and Numerical Precision → explain the shared mechanisms.

The context extension follows the same cost logic. Repack the existing mixture at longer lengths to limit distribution shift, then adapt late instead of paying the final context cost throughout pretraining. The reported bulk of long-context NLL improvement arrives in the first 1–10% of extension, without material short-context degradation in those experiments. Calibration of existing representations is the authors' interpretation, not a universal guarantee. MAI technical report ↗

GLM-5: convert dense attention to sparse retrieval

GLM-5 reports 744B total and 40B active. The checkpoint has 78 decoder blocks—three dense FFNs and 75 MoEs—plus one parameterized MTP module. The prose's “80 layers” is not reconciled by the table/config. The full Q/K head is 256 dimensions: 192 NoPE plus 64 RoPE, not 192 total.

At width 6,144, MLA compresses queries to 2,048 and KV to 512 plus 64 rotary dimensions. Sixty-four heads expand latent keys/values. The DSA indexer uses 32×128-dimensional queries, a shared key, ReLU scores and learned head mixing, then chooses top 2,048 causal positions.

Main attention becomes O(Lk), but the cheaper indexer still scans O(L²) pairs. Sparse retrieval is not mathematically lossless. Conversion trains a frozen-backbone indexer for 1,000 steps, fourteen 202,752-token sequences per step, then jointly adapts for 20B tokens at LR 10⁻⁵. The warm-up sees about 2.839B token positions; it does not perform full-backbone training on all of them.

Each MoE has top-eight/256 plus one shared expert of intermediate width 2,048. Sigmoid affinities weight outputs; balancing bias affects selection, with scale 2.5. Expert matrices alone are about 727B stored and 25.5B active across 75 blocks. The remaining architecture and counting convention explain why those are not the headline totals.

The 27T main exposure is followed by 1T/500B/50B at 32K/128K/about 200K. Repository context includes roughly 10M issue–PR pairs and 160B unique retained tokens. Fuzzy code dedup reportedly adds 28% unique code tokens. Muon Split treats attention up-projections independently by head; cosine/batch warmup and separate sparse-conversion learning rates form part of the recipe. GLM-5 source ↗

GLM-5: trace two independent sparse paths

A decoder block first applies pre-RMSNorm and attention, then adds the residual. A second pre-RMSNorm prepares the FFN, followed by another residual addition. Blocks 0–2 use dense SwiGLU with intermediate width 12,288. Blocks 3–77 use MoE. Every block uses MLA with sparse token selection. Attention sparsity selects past positions; MoE sparsity selects feed-forward networks.

The separate 64-feature rotary key is shared across heads. Each complete query/key head joins 192 non-rotary features and 64 rotary features. Interleaved RoPE rotates only the final slice, with base 1,000,000 and the default configured rule. Sixty-four configured KV heads do not imply ordinary independent full-width KV projections.

The key branch maps the width-6,144 hidden state to one width-128 indexer key, then applies LayerNorm. Indexer heads contain 64 rotary and 64 pass-through features. A learned token-dependent scalar mixture combines the 32 rectified head scores before causal top-k selection. Selected positions cannot include future tokens.

Main value aggregation scales with the selected-key budget, but the smaller indexer still scans all causal pairs. The reported 1.5–2× reduction in long-sequence attention compute follows adaptation; it is not exact equivalence to dense attention. During RL, the team freezes the indexer and uses a deterministic selection path to avoid training/rollout mismatch. Naming torch.topk alone does not guarantee portable tie ordering across devices.

The noaux_tc correction bias changes selection, while original sigmoid affinities determine normalized mixture weights scaled by 2.5. Final RMSNorm precedes an untied 154,880-way vocabulary head. The checkpoint sets BF16 dtype, RMSNorm epsilon 10⁻⁵, no attention bias, zero attention dropout and 202,752 maximum positions. Configured capacity is not a guarantee of retrieval quality. The release uses an MIT license.

Attention → and MoE → own the general mechanisms. Checkpoint configuration ↗ · MLA/DSA implementation ↗ · GLM-5 report ↗

GLM-5: connect data, schedules and execution

Code and reasoning receive early emphasis. Learned web-quality and world-knowledge classifiers supplement mathematical/scientific filtering and language-specific classifiers for lower-resource languages. Natural long documents and synthetic long-range dependency tasks serve different context needs. Repository data joins files, commits, issues, pull requests and source state: the model can connect a reported problem to a changing repository and its repair, rather than see isolated code alone.

Muon Split orthogonalizes MLA Q/K/V up-projections independently by head. The reported comparison matches GQA-8 while keeping attention logits stable without clipping. This partitions update geometry, not merely a scalar learning rate. Head-wise optimization →

Interleaved pipelines can place the MTP module flexibly while sharing the main output head. Pipeline-aware ZeRO-2 uses rolling full-gradient buffers. Communication avoids redundant Muon synchronization. Layer-granular CPU activation offload, selective recomputation and sequence-chunked vocabulary loss reduce different memory terms; deferred weight gradients alter scheduling.

Workload-aware sequence reordering, dynamic attention redistribution and flexible context groups reduce imbalance. Hierarchical expert all-to-all handles routed-token exchange. These are execution dependencies, not evidence that one generic parallel mesh reproduces the run. Distributed Training →

SFT uses up to 202,752 context positions and executable agent trajectories generated with expert models and rejection sampling. Reasoning can be emitted before replies or tools, retained across turns, and controlled per turn. Recoverable errors can remain in the prefix with their loss masked. Masking prevents imitation of the mistake while allowing supervision of its correction.

The asynchronous agent learner records behavior probabilities with exact sampled token IDs. It uses double-sided importance treatment, policy-age and invalid-run filters. Incomplete groups are padded only when enough valid members survive; otherwise the group is dropped. Missing outcomes are not genuine task failures. Periodic weight synchronization and optimizer reset after rollout-engine updates are part of the operational recipe.

Rollout-ID-aware cache hashing, FP8 inference, MTP and prefill/decode separation support more than 1,000 concurrent trajectories. Environment tasks span Python, Java, Go, C, C++, JavaScript, TypeScript, PHP and Ruby. Docker/Harbor terminal builds, the search graph and static/runtime/perceptual slide checks provide distinct feedback. Agent Training → GLM-5 report ↗

GLM-5: separate weights, draft steps and harness results

One parameterized MTP module shares weights across three training unroll positions. Four speculative steps yield reported average accepted length 2.76 versus 2.55 for DeepSeek-V3.2 in that setup. This is not a measured universal throughput multiplier. SFT includes INT4 quantization-aware training with a reported bitwise-identical offline quantization/training kernel; generic BF16 downloads do not automatically provide that path.

CCBenchV2 gives a less uniform picture: 42.1±1.21 resolved on SWE-rebench January 2026,52.3 on chained tasks and 25.8 pass@1 on the backend benchmark. These measures are not interchangeable. A strong open-weight result does not mean every proprietary reference loses on every task.

BrowseComp starts at 55.3. Keeping the latest five interaction rounds and folding older tool observations raises it to 62.0. Adding a discard-all rule beyond 32K raises it to 75.9. The weights did not become a new architecture between these context-policy settings. Tailored prompts, custom harnesses, repaired subsets and model judges constrain cross-model interpretation.

GLM-5.2 later extends configured context to 1,048,576 and introduces four-layer IndexShare from the 128K continued-training stage. Only integer selected-position lists are shared. Every layer retains its own queries, weights, values, output, residual update and MoE. A full indexer is not full dense attention.

KVShare retains target-produced KV across later draft steps instead of mixing target and draft states. The final gain is about 20%, but the cumulative table cannot isolate IndexShare. The reported 2.9× per-token FLOP reduction at 1M is not 2.9× throughput or proportional cache savings. Speculative Decoding →

The original recipe changes architecture, data, scale, post-training and harness together. Internal data and environments limit independent reproduction. Sparse computation still requires storing or distributing all expert weights. Keep these costs and empirical retrieval limits attached to the results. GLM-5 report ↗ · Model card ↗ · Official repository ↗ · GLM-5.2 release ↗ · IndexShare explanation ↗ · IndexCache ↗

GLM-5: train and consolidate agent specialists

GLM-5 follows SFT→reasoning RL→agentic RL→general RL→cross-stage OPD. SFT can retain erroneous context with that segment's loss masked, teaching later recovery without imitating the mistake. Reasoning can precede answers/tools and persist across turns.

Reasoning RL uses group 32, batch 32, binary outcomes, no KL penalty, lower/upper PPO ε 0.20/0.28 and a likelihood-ratio cutoff of two. Clipping still constrains learning. Deterministic top-k and frozen DSA indexing prevent discrete training/rollout mismatch.

Agentic RL runs more than 1,000 concurrent trajectories through slime/TITO. Exact token IDs, behavior log-probabilities and version metadata support importance corrections. Retokenizing text can invalidate that record. Staleness, invalid runs and incomplete groups have explicit filtering; consistent hashing preserves cache locality.

Environment breadth includes more than 10,000 repository tasks across nine languages, terminal containers with over 90% build success, a search graph exceeding 2M pages, and slide grading through static/runtime/perceptual checks. Executable states and verifiers are part of the data.

Cross-stage OPD uses student rollouts scored by selected specialist teachers; teacher–student token log-probability gaps replace ordinary outcome advantages. Group one and batch 1,024 differ from the reasoning-RL batch. This consolidates behaviors without parameter averaging or a fixed teacher-trace corpus.

Infrastructure includes interleaved PP, pipeline-aware ZeRO-2, deferred weight gradients, CPU activation offload, sequence-chunked vocabulary loss, flexible context groups and hierarchical expert all-to-all. INT4 QAT during SFT and a reported identical quantization/training kernel are not automatic properties of downloading a BF16 checkpoint. GLM-5 source ↗

GLM-5.2 and 5.3 optimize different bottlenecks

GLM-5.2 IndexShare reuses selected token positions across four neighboring sparse-attention layers. Each layer still forms its own queries, weights and aggregation. Continued training introduces the sharing pattern; an inference-only switch is not equivalent.

The reported 2.9× per-token FLOP reduction at 1M is not 2.9× end-to-end throughput or cache savings. MTP IndexShare/KVShare plus later rejection sampling and total-variation loss raise accepted length cumulatively from 4.56→5.10→5.29→5.47. The table cannot isolate one ingredient.

GLM-5.3-Flash instead compresses indexer candidates within a layer. Learned per-dimension pooling gates combine four keys. The selector ranks up to 512 pools, expands them into 2,048 original positions and appends up to three incomplete-tail keys. Main attention reads raw KV, not pooled values.

IndexPool does not quarter main top-k attention. Its generic implementation can retain raw keys/gates, so cache compression needs the optimized storage path. Its full indexers do not use the regular four-layer IndexShare pattern. GLM-5.2 release ↗ · GLM-5.3-Flash source ↗

GLM-5.3-Flash: what the implementation discloses

This is a new 320B/18B native multimodal model, not a compressed GLM-5.2 checkpoint. Its 45-block stack has 34 KDA and eleven NoPE MLA/DSA blocks, first three dense FFNs, then top-eight/288 plus one shared expert. Approximately 305.5B expert matrices account for most storage; 9.51B active expert matrices do not include the rest.

KDA expands width 4,096 into 64 heads of width 128, uses causal kernel-four convolution, normalized Q/K, bounded log decay and feature gates. NoPE MLA uses query rank 1,536, KV rank 512 and 64×256-dimensional heads. The text stack has no rotary slice.

Four-stream mHC wraps attention and FFN separately. Learned pre/post maps collapse and distribute sublayer signals; a 4×4 combine matrix uses twenty Sinkhorn iterations. Streams persist until the final mean. That costs residual traffic; it is not inherently an inference optimization.

The 24-block vision encoder uses width 1,024, sixteen heads, 2×14×14 temporal/spatial tubelets and a 2×2 spatial merger. A nominal 448-square image gives $32^2$ patches then $16^2=256$ language tokens per tubelet, before actual processor resizing. Vision attention remains noncausal and uses positional rotation even though text MLA is NoPE.

The release claims 30T multimodal pretraining, but omits the full mixture, optimizer, schedule and compute. Render–inspect–revise visual coding and user-flow verifiers are useful post-training ideas without disclosed dataset/reward ablations.

Serving separates Encode, Prefill and Decode; recurrent replay, W8A8 and mixed cache precision address different workloads. The reported 3× improvement is against the initial same-hardware implementation. Analytical 3× attention / 4.4× KV comparisons are not independently reproduced end-to-end results. GLM-5.3-Flash source ↗

GLM-5.3-Flash: forget, correct, then read

Kimi Delta Attention (KDA) compresses the prefix into a fixed-size matrix. A width-four causal depthwise convolution captures short local structure. The matrix can retain associations from the whole prefix; its limitation is compressed retrieval, not strictly local access.

GLM uses 64 heads of width 128. Q, K and V each expand the width-4,096 input to 8,192 features. Causal convolution and SiLU prepare those features before the recurrent kernel. A scalar write gate applies per head and token; a feature-wise output gate acts after the readout.

The reference kernel casts recurrence arithmetic to FP32. It normalizes Q and K with an additive epsilon, then scales the query. Define one head's state with key features as rows and value features as columns.

The example holds key, query, value, decay and write gate fixed. The keys and queries supplied to this example are already normalized. It is a synthetic calculation, not a measured memory trajectory or learned GLM activation. All four operations remain available through the slider without playback.

The configured log-decay lower bound is −5. Bounding one step helps reciprocal rescaling, but does not bound a product over arbitrarily many steps. Chunk length and accumulation precision remain important. After readout, RMS normalization and sigmoid feature gating precede flattening and projection back to width 4,096.

Periodic NoPE MLA/DSA blocks recover precise retrieval from selected raw KV entries. Their query rank is 1,536, KV rank 512, and 64 heads have 256 key and value features with zero rotary dimensions. Both latent projections pass through RMSNorm before expansion. Queries expand to 64 × 256; each head's KV expansion has 256 key plus 256 value features. Causality and KDA recurrence still supply order information. Attention → connects recurrent compression to sparse retrieval. GLM5-Next reference implementation ↗

GLM-5.3-Flash: select pools, read original tokens

Sparse attention avoids aggregating every previous value, but its lightweight indexer still scores every candidate. At long context, that quadratic selector becomes expensive. IndexPool groups four consecutive indexer keys before scoring.

Each token provides a 128-feature key and 128 pooling logits. A learned four-position embedding modifies the logits. Softmax runs across the four members independently for each feature.

The first feature gives the final member three times the unnormalized mass of each other member. Its pooled value is 8.0. The second feature uses equal weights and gives 6.0. One scalar weight per token cannot express this difference. The example combines gate and position logits; it is not a learned model observation.

The selector uses 32 query heads of width 128, ReLU query–pool scores and learned head mixing. It chooses up to 512 complete pools, then expands them to 2,048 raw token positions. Main MLA attention retrieves original KV entries, not pooled values.

This demonstration selects one complete pool. Fixed synthetic scores are 1, 3 and 2; a higher score wins among visible complete pools. Advancing the slider reveals tokens without changing scores. At six tokens, pool zero expands to positions 0–3 and tail positions 4–5 are appended. At eight tokens, pool one becomes selectable and wins: positions 4–7 replace 0–3. The demonstration shows selection and causality, not actual model relevance.

The released budget permits up to three visible tail tokens in addition to 2,048 expanded positions: at most 2,051. A pool cannot be selected before its final member is visible. The implementation also handles padding, invalid slots and static caches. Grouping starts at the first real token; invalid output slots use −1. The small example assumes a contiguous unpadded prefix and does not emulate those cache paths.

IndexPool reduces scored candidate count by roughly four, but does not quarter main attention's top-k. Generic code retains raw keys and gates and rebuilds pools; storage savings depend on an optimized serving path. IndexShare instead reuses selections across depth. These are different optimizations. Sparse attention → supplies the retrieval context. Pooling and causal selection implementation ↗

GLM-5.3-Flash: keep four streams between sublayers

Attention changes access across tokens. Manifold-Constrained Hyper-Connections (mHC) change access across residual streams. Ordinary residual addition carries one vector forward. GLM instead copies each initial text embedding into four streams, keeps those streams between blocks, and learns how each sublayer reads and updates them.

1. Keep: input X has four rows of width 4,096 per token.
2. Collapse: learned pre-weights form one width-4,096 input. RMSNorm prepares that input for attention or FFN.
3. Compute: the sublayer produces one width-4,096 output.
4. Merge: learned post-weights distribute the output. A separate combine matrix mixes the old streams.
5. Repeat: the FFN has its own mHC wrapper. The next block still receives four streams.

The combine matrix is constrained toward a nonnegative matrix with row and column sums near one. This limits amplification along the old-stream mixing path. It is not a proof that learned pre/post gates or the entire network remain norm-preserving.

The implementation uses twenty Sinkhorn normalization iterations and epsilon 10⁻⁶. Finite iterations and epsilon make the constraint approximate. The four streams are averaged only at the decoder end, before final RMSNorm and the vocabulary projection. Four-stream state increases residual traffic; mHC is not inherently a serving-speed optimization.

mHC changes residual mixing; Attention Residuals instead retrieves earlier depth representations. Depth access → keeps that distinction separate. NoPE → explains the text position choice. GLM-5 and Kimi K3 have their own recipe sections below; their reports are not substitutes for this model's configuration. mHC paper ↗ · Reference stream orientation and collapse ↗

GLM-5.3-Flash: sparse work does not remove stored experts

Blocks 0–2 use dense clamped SwiGLU FFNs: width 4,096 → intermediate width 12,288 → width 4,096. Blocks 3–44 use MoE FFNs. Each stores 288 routed experts and one shared expert, with intermediate width 2,048. A token executes eight routed experts plus the shared expert.

The router calculates logits in FP32 and applies sigmoid. Correction biases affect expert selection, not output weights. The selected original sigmoid scores are normalized and scaled by 2.5. MoE → explains why selection scores and mixture weights must stay distinct.

The gate branch has an upper clamp of 10; the up branch is clamped to [−10, 10]. These controls limit extreme intermediate activations. They do not replace the routing balance rule. Attention, routers, mHC, embeddings, vision and auxiliary modules explain why expert-only counts differ from the model headline. Do not assume that every auxiliary module executes on every ordinary token.

The configured vocabulary has 154,880 entries; input and output embeddings are untied. Text attention has no bias and zero dropout. There is one configured MTP module and a maximum position setting of 1,048,576. That setting alone does not establish reliable use of every position.

The text dtype setting is BF16, but the current release configuration also contains an FP8 E4M3 policy with dynamic activations and excluded modules. Treat precision as module- and export-dependent, not as a uniformly BF16 resident payload. The source reports an MIT release license. Released configuration ↗ · Precision →

GLM-5.3-Flash: encode visual tokens before language execution

Native vision adds a distinct dense workload before language prefill. The 24-block encoder has width 1,024, sixteen width-64 attention heads and FFN width 4,096. A 3D convolution embeds temporal/spatial tubelets of 2 × 14 × 14.

Vision blocks use noncausal full attention, Q/K normalization, rotary position information, pre-RMSNorm and clamped SwiGLU. Text NoPE does not mean vision has no positional encoding. Image/video packing and processor resizing still determine the actual sequence geometry.

1. Encode: run the 24 vision blocks at width 1,024.
2. Normalize and merge: final RMSNorm precedes a strided 2 × 2 spatial convolution, producing width 4,096.
3. Project: the merger processes that vector through a gated width-10,240 intermediate and returns width 4,096.
4. Inject: replace matching image/video placeholders in the text embedding stream.

Encode–Prefill–Decode (EPD) places visual encoding, prompt processing and autoregressive decoding in separately scalable worker pools. Dense visual encoding, compute-heavy prefill and memory/communication-heavy decoding have different resource needs. Serving Models → explains the scheduling and transfer costs.

The release also lists intra-node tensor parallelism for KDA and the language head, ReplaySSM, W8A8, mixed INT8/FP8/BF16 cache precision and Layer Split. These are co-designed serving ingredients, not properties guaranteed by a generic eager checkpoint loader. Eager dense masking can retain much of sparse attention's overhead.

The reported deployment spans tens of thousands of Chinese accelerators. The reported 3× improvement compares with the provider's initial implementation on the same hardware, not with every competing model. IndexPool still trades fine-grained selector precision for compression. Low active compute still leaves 320B-scale storage and expert communication; native vision adds encoder work and visual-token prefills.

No controlled result here proves that combining IndexPool with depth-wise IndexShare would improve a different model. That is a possible design experiment. Training mixture, visual RL and evaluation limits belong to the following evidence section. Vision and token-injection implementation ↗ · Kimi Linear ↗ · ReplaySSM ↗

GLM-5.3-Flash: inspect the whole comparison

Reduced active computation is useful only if the resulting model preserves the capabilities the workload needs. Start with the base checkpoint: it precedes the agent interface and post-training changes that can dominate deployed comparisons.

Flash improves the reported coding-base score, but scores lower on all four listed general or factual benchmarks. This is a capability tradeoff, not uniform domination. It also does not isolate the hybrid attention design: data, scale and optimization can differ between these runs.

Post-trained comparisons answer a different question: how well does the complete model, tools and harness solve the task? The comparator columns expose workload differences that disappear if we select only each model's best result.

For example, Flash exceeds the listed GLM-5.2 scores throughout this subset. Against Claude, the directions differ: DeepSWE, Toolathlon and AutomationBench favor Flash, while Terminal-Bench, NL2Repo, Agents' Last Exam and HLE favor Claude. A single mean would hide those workload differences.

The release uses Claude Code for Terminal-Bench and mini-swe-agent for DeepSWE. NL2Repo uses a 1M context; HLE includes tools and context management. Other differences include sampling, context limits, model judges and tool access. These conditions prevent an architecture-only reading of the table.

The model card specifies maximum reasoning effort for reproduction. Terminal-Bench uses Claude Code 2.1.207, temperature 1, top-p 1, 65,536 maximum new tokens and a six-hour timeout. DeepSWE uses temperature 0.95, top-p 1, 400K context and a six-hour timeout. NL2Repo uses temperature 1, top-p 1 and 64K maximum new tokens. HLE uses temperature 1, top-p 0.95, 163,840 maximum generated tokens, 300K context and GPT-5.6-luna (medium) judging. Toolathlon reports pass@1 averaged across three independent runs. AutomationBench v1.0.6 includes the null-type handling fix. These are release-specific contracts, not recommended settings for every task.

Visual scores need their own task contract

For multimodal tasks, the release reports OfficeQA Pro 62.4, CharXiv Reasoning with tools 89.4, Chartography with tools 78.0 and MVBench 77.8. These are four different evaluations, not four estimates of one general visual ability. Tool-assisted reasoning is not directly interchangeable with a no-tool perception score.

Visual coding adds a render–inspect–revise loop: code produces an artifact, the agent inspects that artifact, and environment feedback guides another edit. User-flow verification can test layout, interaction behavior, GUI judgment and task success beyond compilation. The release does not disclose dataset size, environment count, rollout volume, reward formula, verifier accuracy or a controlled ablation for this loop.

Keep operational claims separate from quality

The provider's 3× end-to-end serving improvement compares with its initial implementation on the same hardware. Its 3.0× attention-compute and 4.4× BF16 KV-cache comparisons against GLM-5.3 use architectural accounting, without a complete reproducible formula table. Neither is a universal cross-model latency guarantee.

A “one-tenth the price” product claim is also a snapshot. Utilization, batching, cache hits, quantization, hardware and subsidies can change price without changing the checkpoint. Low active parameters do not remove 320B-scale storage or expert communication costs.

The 30T multimodal training claim lacks a disclosed modality mixture, deduplication recipe, optimizer, schedule and compute budget. Released weights and code can establish implementation structure, but cannot supply these missing run details. Z.ai release ↗ · Model card and weights ↗

Kimi K3: compressed sequence and depth access

Kimi K3 combines 2.78T total with 104.2B active, 93 layers at width 7,168, 96 attention heads, a 160K vocabulary, 1M context, and top-sixteen/896 routed experts plus two shared experts. Routed selection is one expert in 56, not an overall compute ratio.

Sixty-nine KDA layers maintain recurrent memory; twenty-four gated MLA layers retain precise global retrieval. Lower-bounded log decay $g\in(-5,0)$ gives $\alpha=e^g\in(e^{-5},1)$ in the analytic finite-input rule. This limits reciprocal-rescaling range for efficient BF16 kernels. Gates and numerical rounding still need the implementation contract.

Attention Residuals selects normalized depth keys while mixing unnormalized values. Eight blocks include a partial final block, plus the embedding source. The report confirms this layout; it does not imply 96 full decoder layers.

LatentMoE routes at width 3,584 and projects back; shared experts keep full width. Pre-up-projection RMSNorm, bounded SiTU-GLU branches with $\beta_1=4,\ \beta_2=25$ and quantile-based balance control limit sparse-path growth. The narrower space alone does not establish stability.

FlashKDA, associative context parallelism, MoonEP redundant experts/plans, activation recomputation/FP8/offload and hybrid cache ownership support the architecture. NoPE avoids a rotary interpolation rule but does not make order or long-range retrieval free. The reported 2.5× scaling efficiency belongs to the combined recipe. Kimi K3 source ↗

Kimi: decay the memory before correcting it

A KDA head stores a key-to-value matrix instead of a list of past tokens. Its query and key have width dₖ; its value has width dᵥ. Short convolutions and Swish precede query/key L2 normalization. The write strength β is a sigmoid output. Forgetting acts separately on each key feature.

The operation order matters. With a scalar state of 2, decay 0.5, key 1, value 3 and $\beta=1$, decay gives 1. The correction is $3-1=2$, so the final state is 3. Predicting from the old state would give 2 instead.

K3 bounds log decay to protect the reciprocal cumulative factors used by its chunked kernel. Its learned decay logits use a low-rank projection and head bias. The separate per-head A starts at zero.

The tile bound keeps reciprocal scaling within BF16 dynamic range. It permits a Tensor Core diagonal-tile path rather than expensive position-pair calculations. BF16 representability does not remove rounding error. The chunked causal lower triangle includes its diagonal because the read follows the current write.

The fixed matrix compresses history. Twenty-four gated MLA layers still retrieve particular earlier tokens. NoPE removes the need to rescale rotary positions, but it does not guarantee perfect long-context retrieval. Attention → owns the broader recurrent/softmax comparison. Kimi K3 report ↗ §2.1.

Kimi: choose representations across depth

Ordinary residual addition merges earlier updates into one accumulated state. Attention Residuals instead choose among earlier updates. A learned query belongs to the receiving layer; the keys are normalized source representations. The values retain their original scale.

Full depth attention repeatedly reads an increasing number of sources. K3 sums updates within each block, then attends over completed block summaries and the current partial block. The current partial source is absent at the first layer of a block. The embedding remains available.

The primary report explicitly specifies eight blocks with block size twelve and a partial final block, plus the embedding: nine sources in total. This is not a claim that the decoder contains 96 full layers. Learned depth selection and compressed sequence memory solve different problems.

The MoE then provides selective parameter access. Its routed path uses width 3,584, RMSNorm before the up-projection and bounded SiTU-GLU with $\beta_1=4,\ \beta_2=25$. Two shared experts retain full width. Quantile Balancing computes biases from score-margin quantiles, rather than a fixed-size signed correction. Narrowing the path alone does not establish stability.

Block Attention Residuals → explains the summaries. LatentMoE → contains the projection flow and exact activation formula. Quantile Balancing → follows the controller. Kimi K3 report ↗ §§2.2–2.3.

Kimi: connect visual inputs, data and task dependencies

The architecture needs a data recipe that exercises its memory and selective capacity. MoonViT-V2 has about 401M parameters and 27 layers. It trains jointly from scratch with next-token prediction, rather than starting from a contrastive SigLIP checkpoint. Images and video share parameters. The encoder uses 14×14 patches and factorized spatial/temporal attention. A 2×2 pixel shuffle reduces visual tokens fourfold; maximum image resolution is 3,584×3,584.

The authors report matching their SigLIP-initialized baseline with lower gradient norms and fewer spikes. This result does not establish that contrastive initialization is always unnecessary. The mixture includes natural and interleaved images, OCR, perception, video, visual code and rendered SVG, web, game, 3D and CAD artifacts. These sources exercise interfaces and documents as well as natural-image recognition.

Text pretraining combines web, code, mathematics and knowledge. Fidelity-checked rewrites vary style and perspective. The team tunes cosine and Warmup-Stable-Decay schedules independently before comparison; cosine wins in that setting. Sharing one hyperparameter choice can measure tuning bias rather than schedule quality.

Context grows 8K→64K in main training, then 256K→1M during cooldown. Concatenated and permuted multimodal documents and subtasks create dependencies across the sequence. Presenting a long input alone does not ensure that solving it needs distant evidence.

Agent training extends that principle from distant evidence to actions. A self-expanding hierarchical knowledge graph supplies retrieved material, increasingly fine concepts and combinations of related nodes. Autonomous Execution Tasks define initial state, goal, tools, budget and verifier, without a reference trajectory. Configurable harness modules vary system prompts, tools, context management, skills, memory and subagents. Public diagnostics guide recovery; hidden final verification evaluates completion.

This gives specialists tasks that require decomposition, execution and recovery, rather than imitation of one tool sequence. The next section connects those tasks to rewards and consolidation. Kimi K3 report ↗ §§2.4,3–4.2.

Kimi: consolidate specialists on student prefixes

The recipe first trains SFT, then domain/effort specialists, then consolidates them. Specialist families cover general reasoning and vision, search and knowledge, long-horizon agents, and coding. Nine teachers do not imply every Cartesian combination of family and effort.

The student receives guidance on contexts it actually visits. The system need not generate a large fixed dataset of teacher trajectories for this stage. Specialists can improve independently before consolidation. The report found no clear benefit from its tested top-k distillation objectives; that is a setting-specific observation.

Training begins with a large but capped max-effort budget. The team lowers τ to obtain high- and low-effort teachers, with domain-specific human guidance. This training consequence is neither a soft length penalty nor an inference hard-token cap.

Non-verifiable tasks also use an agentic generative reward model. Its judge reads the artifact, creates a rubric, scores candidates and records scores before tournament-style binary comparisons. A candidate beyond its cold-start verbosity multiplier automatically loses. This output-verbosity rule is distinct from the reasoning budget.

Partial rollouts address stragglers. Generation pauses when a fraction of the N×K trajectories finishes. Unfinished runs are prioritized next iteration. A prompt group reaches optimization only once all K responses complete. Resumption can cross policy versions; per-token regularization keeps updates local, but does not make stale tokens on-policy.

Distillation →, Reasoning Effort → and Agent Training → own the reusable mechanisms. Kimi K3 report ↗ §§4.1.2–4.1.3.

Kimi: align the deployment objective and read the scores

Deployment-aware post-training uses MXFP4 routed expert weights and MXFP8 expert activations throughout SFT and RL. Attention, latent projections, shared experts and routers retain higher precision. Matching quantization reduces one training/rollout mismatch; it does not prove every runtime operation is identical.

A pretrained MTP layer becomes a one-layer EAGLE-3 drafter. The target stays frozen; only the draft layer and its feature-fusion projection train. The drafter joins outputs from the first, fourth and final AttnRes blocks. Its bias-free fusion matrix starts as [0 0 I], so initialization keeps the high-level feature. Seven training unrolls feed earlier draft outputs after the first step.

Distribution overlap is the expected single-token acceptance under lossless speculative sampling. It is not itself throughput: proposal work, verification and state repair still cost time. KDA cannot remove a rejected suffix by slicing its matrix. K3 caches projected draft inputs and replays only the accepted prefix. Speculative Decoding → retains acceptance and cache-correctness conditions.

Single-step evaluations use top-p 0.95; agentic evaluations use 1.0. Terminal-Bench reports the best score across harnesses for each model. BrowseComp uses compaction at 300K tokens; without context management over the full 1M window, K3 scores 90.4 instead. Vision scores generally average three runs. Tools and harnesses are part of these results, not cosmetic settings.

Some tasks use model judges or internal environments. External leaderboard results are dated snapshots. The combined recipe changes architecture, data, scale, RL and systems, so these scores cannot isolate one component’s contribution. The reported 2.5× scaling efficiency is likewise a complete-recipe comparison, not a measured 2.5× serving speedup.

FlashKDA handles the recurrent operator; associative context parallelism joins segment transitions. MoonEP uses redundant experts and routing plans. Recomputation, FP8 activation storage and remote offload reduce different memory terms. Checkpointable AgentENV microVMs and the CPU KV pool let waiting trajectories release active resources. Its reported 51M sandboxes and 1.5M images are accumulated infrastructure counts, not unique tasks or simultaneous concurrency.

The systems follow the model’s dependencies: fixed recurrent state, growing global KV, distributed expert weights and suspended environments. Distributed Training → and Serving → explain these separate costs. Kimi K3 report ↗ §§4.1.4–6 · Kimi release explanation ↗ · Released weights ↗

DeepSeek: change the prefill dependency

DeepSeek-V4.1's 552B backbone plus 196B Engram has twenty causal encoder and twenty decoder blocks. Decoder global KV comes from final encoder states. Long-prompt prefill therefore mostly uses the lower half, while generated tokens use all forty blocks: 8B active prefill, 16B decode.

The bounded decoder-tail replay approximates local state, not a full-depth prompt computation. Post-training simulates that approximation, but uncharacterized edge cases remain. Encoder prefix reuse has its own bounded replay.

CSA2 Full constructs global state/selection, Reindex reuses state with fresh selection, and Reuse also reuses selection. Main queries and local 128-token windows remain layer-specific. The encoder has two SWA-only blocks then three six-block sharing groups; the decoder has an initial four-block group then four Reindex-led groups.

The first decoder selector chooses 512 entries and up to 2,048 eight-token blocks; later scans choose within that 16,384-entry pool. The initial scan still scales with context, and excluded evidence cannot be recovered downstream. Compression one means uncompressed positions, not dense selection.

After-RoPE FP4 global cache with E4M3/16-channel scales, local FP8, QAT and sharing yield a reported 890-byte/token global component. It is reconstructed before attention. Two Engram modules use n-grams two/three/four and hash-based lookup; deterministic addresses support host prefetch. Single-Pass mHC shifts mixing coefficients for fusion, halving one activation-traffic term, not all traffic. DeepSeek-V4.1-Flash source ↗

DeepSeek: separate local state, shared KV and lookup memory

The causal encoder–decoder changes a dependency, not merely a kernel. In an ordinary stack, each layer needs the preceding layer’s prompt representations to build its cache. Here the decoder projects global memory from final encoder states. Future tokens remain inaccessible: the encoder is causal, not bidirectional.

Each decoder layer still needs its own sliding-window state. Serving replays the last 128 encoder positions through the decoder, restricting local attention to that segment. It does not persist decoder SWA state between prefills. Encoder prefix-cache reuse likewise reconstructs local state from a bounded tail while retaining global state. Stacked local layers can depend on positions before the replay segment. This approximation is simulated during post-training; a reported small quality effect does not eliminate uncharacterized edge cases.

Encoder: 2 SWA-only + 3 × [Full, Reuse, Reuse, Reuse, Reuse, Reuse]
         Global compression ratio 2
Decoder: [Full, Reuse, Reuse, Reuse] + 4 × [Reindex, Reuse, Reuse, Reuse]
         Global compression ratio 1; still sparse selection
Every layer retains its own 128-position local attention path.

Compared with earlier CSA, compression uses nonoverlapping groups of m tokens, rather than overlapping groups of 2m. It removes the compressor’s absolute-position component. Indexer keys come from main KV instead of an independently compressed path.

The first decoder Full layer scans the global context and chooses its own top 512 entries plus the shared candidate pool. Later Reindex layers choose 512 only within that pool. The initial scan still grows with sequence length. Excluded evidence cannot reappear downstream. The hierarchy is introduced during post-training, not only at deployment.

Main global KV uses E2M1 values and one E4M3 scale per sixteen channels, without NVFP4’s second-level global scale. Quantization occurs after RoPE and uses post-training QAT. More sensitive local SWA cache stays FP8. Global values are dequantized before attention; this is not a claim of native FP4 attention matrix multiplication. The reported 890 bytes per input token becomes 890 MB decimal, about 0.83 GiB, at one million tokens. That excludes local caches, activations, workspaces and model weights.

Engram adds a different memory. Two modules at zero-indexed layers 1 and 14 split 196B parameters evenly. Each uses orders 2, 3 and 4, eight hash heads and total embedding width 2,048 per order. Heads address roughly 16M entries with distinct prime table sizes. Tokenizer compression, hashing, context-aware gates and residual integration remain; the short causal convolution is omitted because its measured benefit did not justify serving complexity.

Tables and key/value projections use FP8. Deterministic addresses permit background RDMA prefetch from host memory when serving. RL rollouts instead keep tables on GPU to reduce host pressure and fragmentation failures. These placements are not interchangeable memory estimates.

The serving Mega-mHC kernel fuses residual update, input mixing, coefficient prediction, pre-normalization and FP8 conversion. Pretraining keeps multiple kernels with the same changed coefficient dependency. Attention →, Conditional Memory → and Serving → explain the connected operators. DeepSeek V4.1 report ↗ §§2–3 · YOCO antecedent ↗.

DeepSeek: stage vision and parameter-specific updates

The 45T exposure mixes text-only and multimodal tokens 7:1. Native image–text/interleaved/PDF/grounding/OCR/image-code data pass cheap filters before costly retrieval, then visual scoring/deduplication. Cross-representation dedup replaces matching text with multimodal content; deterministic splitting/packing reports padding at most 10⁻⁴.

Vision first uses a SigLIP-style 47B-pair bootstrap at up to 224 pixels. An auxiliary 4B MoE then supplies 236B generative-alignment tokens at 544–1,344 pixels; the auxiliary LM is discarded. The main integration initially freezes vision except final norm, trains the projector, and later unfreezes at smaller LR. Higher-resolution bootstrap contributed little after later stages in this comparison.

Text and image routing have separate correction biases. The 32-layer vision encoder uses 2D RoPE/RMSNorm/SwiGLU; 3×3 pixel unshuffle reduces language-side image tokens ninefold.

Global batch is 100.6M tokens; sparse attention starts at 64K and extends to 1M at 34T. LR warms 2,000 steps, stays 2.6×10⁻⁴ until 28T, cosines to 2.6×10⁻⁵ by 40T, then stays constant to 45T. About 447K steps is a constant-batch calculation, not GPU-hours.

Muon handles backbone/projector/Engram projections, with head-wise Q/K preconditioning; nonmatrix parameters use AdamW. Tables, embeddings and head use Nesterov plus eleven Sinkhorn balancing iterations without weight decay, $\tau=10^{-3}$, $\epsilon=10^{-20}$. This balances update magnitudes, not mHC's forward mixing matrix.

Sharing across PP boundaries requires one logical owner, shadow indexers, synchronized parameters and microbatch-scoped payloads. Engram is row-sharded/prefetched; vision work is separated and images balanced across context ranks. DSpark trains after backbone pretraining under a frozen backbone, then tracks post-training without sending its loss into backbone weights. DeepSeek-V4.1-Flash source ↗

DeepSeek: retain the data and optimizer contract

The 45T exposure has a 7:1 text-only/multimodal token ratio. This is neither a document ratio nor a count of unique tokens. Scaling experiments vary data and model size. Domain experts define finer quality criteria. Filtering targets low-information generated text, weak-model outputs and poor machine translations, not every form of synthetic data. Coding adds recent repositories, commits, libraries, frameworks and languages; exact language/code allocations remain undisclosed.

Image–alt-text relevance filtering and semantic image deduplication complement interleaved webpages and PDFs. Grounding, pointing, OCR, image–code pairs and computer-use trajectories supply other tasks. Cheap document checks precede costly image retrieval. Image-aware filtering and SmolVLM scoring follow. Some rejected interleaved documents become image–text pairs. Renewed crawling addresses the text bias of existing Common Crawl material.

Cross-representation deduplication replaces matching text with its multimodal version and retains the larger configured epoch count. Pretraining and context-extension allocation are coordinated to reduce overlap. Deterministic splitting and best-fit packing keep reported padding at or below 10⁻⁴. Training Data → owns the reusable pipeline.

Muon updates backbone matrices, Engram projections and the vision-language projector. Q/K weights receive head-wise preconditioning. Nonmatrix parameters use AdamW. The table/embedding/head optimizer balances squared update magnitudes with alternating row/column scaling. It replaces Newton–Schulz on updates, not the forward embedding. Retaining momentum without Adam’s second-moment buffer matters for a 196B table. This optimizer operation is distinct from forward mHC Sinkhorn-Knopp mixing.

The vision encoder has 32 layers, width 1,024, sixteen heads, 14-pixel patches, 2D RoPE, RMSNorm and SwiGLU. Linear patch projection suits Muon. A 3×3 pixel unshuffle moves nine patches into channels before a two-layer width-5,120 bridge. Maximum image resolution is approximately 1,344×1,344. The earlier vision-training section keeps bootstrap, auxiliary 4B alignment, transfer and later unfreezing separate.

Shared attention crosses pipeline boundaries. Shadow indexers retain one logical parameter owner, synchronized weights and gradients, explicit KV/index payloads and microbatch-scoped lifetimes. Engram tables are row-sharded across dedicated groups; optimizer state is further sharded across replicas. Deterministic lookup allows prefetch before a microbatch. Vision executes separately from language forward/backward; large images are balanced over context ranks and loaded once.

At a constant 100.6M-token batch, 45T corresponds to about 447,000 steps. That arithmetic does not disclose GPU-hours or calendar time. Optimization → and Distributed Training → explain update and ownership contracts. DeepSeek V4.1 report ↗ §§3–4.

DeepSeek: make environments and verification scalable

SFT→RL→final full-vocabulary OPD uses more than forty heterogeneous teachers. Intermediate model merges initialize RL runs; final student distillation is a different mechanism. Exact SFT sizes, reward coefficients, teacher allocation and unique task budgets remain undisclosed.

Task construction separates planner, builder, independent solvers, inspector and repair. Real sessions/repositories seed executable tasks with fail-to-pass and pass-to-pass checks. Builders remove solution leakage; inspectors test instruction/verifier consistency and exploits. Constructor rewards target difficulty and correctness, and new solver traces trigger re-audits.

General-agent tasks reconstruct tools and negative feedback from voluntary workflows. Mock schemas are not proof of real-service fidelity. Training spans versions and heterogeneous scaffolds through an external trajectory worker/common schema.

DSec shards resource/failure domains, uses relaxed scheduler estimates with node admission, sub-NUMA placement and isolation policies. More than 2,500 live containers per node versus about 1,000 does not mean that many concurrent repository builds. Priority classes and hyperthread separation protect timing-sensitive grading. Sandbox crashes count as failure; infrastructure and package services can leak answers.

Rollout/training time-share GPUs and interrupt at token boundaries, while environment state lives outside preemptible GPU pools. Sample-level dispatch avoids waiting for whole groups, but relative advantages still belong to the same prompt. Reused cache/routes can span model versions; recorded routes and staleness masks preserve the intended path.

Per-dataset concurrency and optional short-early-return rejection control completion-time bias. Staleness is a different issue. Effort b∈[1,100] sets a capped length penalty whose coefficient falls exponentially with effort; centering stays within problem/effort subgroups. Presets 50/75/100 are not hard token limits. DeepSeek-V4.1-Flash source ↗

DeepSeek: build a task that survives independent checks

A task joins a problem, executable environment and verifier. A plausible instruction is insufficient if the container is broken or the test rewards the wrong behavior. Constructors receive difficulty and correctness feedback. New solver trajectories trigger fresh audits when tasks enter another RL run.

1. Collect. Voluntary employee/partner sessions are filtered for difficulty or weak performance, then deduplicated by trajectory. GitHub repositories above an unspecified star threshold provide another source.
2. Plan. Check buildability and automatic verification. Select a turn or commit, propose substantial implementation work, define target and regression tests, document required external resources.
3. Build. Install dependencies, prepare the tree, write instructions/tests, self-test, remove solution leakage and package a new container-image layer.
4. Solve independently. Several agents expose ambiguity, unsolvability, difficulty and possible exploits.
5. Inspect independently. Check setup, factual consistency, instruction/test agreement and hackable scoring against both artifacts and trajectories.
6. Repair and recalibrate. Fix defects and adjust tasks that are too easy or too hard. Repeat construction/validation when needed; later trajectories feed re-audits.

Fail-to-pass checks cover the requested repair; pass-to-pass checks protect existing behavior. Starting from a commit does not restrict construction to replaying its reference patch. The planner can synthesize new work. Generated tests can still be inconsistent or exploitable, which motivates separation of responsibilities.

General-agent synthesis reconstructs voluntary workflows, schemas, constraints and tool behavior for SaaS, enterprise and specialized backends. Negative feedback seeds single- and multi-turn failure situations. Mock tools require fidelity validation; matching an API schema alone does not reproduce a real service’s failures. Agent Training → and GLM’s environment recipe → provide connected mechanisms, not a claim that V4.1 invented each pattern.

DSec targets millions of concurrent instances. Scale units contain failures and resource exhaustion. Independent schedulers estimate resources with relaxed global consistency; each node makes the final admission decision. Worker VMs bind to sub-NUMA domains, with containers using local CPU/memory. The reported increase from about 1,000 to over 2,500 live containers per node is workload-specific, not simultaneous repository builds.

Latency-sensitive classes stay separate from SCHED_IDLE background jobs. Core scheduling prevents different priorities sharing sibling hyperthreads. AppArmor profiles and fine-grained eBPF network policies limit interference and leakage. Sandbox crashes count as failed trajectories and emit training feedback. Package services and infrastructure remain part of the verifier’s attack surface.

Claude Code, OpenCode, Pi and DeepSeek Harness variants expose different interfaces and versions. A trajectory worker uses a common training schema; tools and environment state live outside the preemptible GPU pool. GPUs alternate rollout and learning, interrupting generation at token boundaries. Completed samples free dispatch capacity without waiting for the same earlier group. Group-relative advantages still belong to one prompt.

Resumed trajectories can reuse KV and recorded expert routes under newer weights. Concatenated routing replay retains each segment’s choices; it does not regenerate history with the latest checkpoint. Dataset concurrency and optional short-early-return rejection address completion-time bias. Dispatch/waiting bounds and excessively stale-token masks address policy age. The undisclosed thresholds must not become invented universal defaults.

Center rewards within the same problem and effort subgroup, not across low/high responses. API presets are 50, 75 and 100; interpolation is not an inference hard-token limit. Intermediate model merging initializes successive RL runs, with its exact rule undisclosed. Final full-vocabulary on-policy distillation consolidates over forty heterogeneous teachers on student-visited prefixes. Teacher/domain allocation, mixture and concurrency can change with evaluations. The complete objective is not disclosed; this is distinct from teacher-trace SFT and intermediate parameter merging. DeepSeek V4.1 report ↗ §5.

DeepSeek: compare the same benchmark and expose missing costs

DSpark has three blocks, five parallel draft positions, a local window of 128 and a lightweight Markov head for draft dependencies. A confidence head estimates acceptance; scheduling combines those estimates with engine throughput profiles. Backbone pretraining omits MTP. A separate frozen-backbone stage trains DSpark, then post-training updates the drafter without sending its loss into backbone weights. Serving and RL/OPD rollouts use it. Speculative Decoding → owns the broader implementation.

The original release check found NL2Repo 64.0 on the model card versus 65.4 in the report. These are separate artifact values. Terminal-Bench 2.1’s high result must not obscure much lower absolute results on versions 3 and 4. ZeroBench 49.0 is pass@5, not pass@1. Native multimodality is not evidence of winning every visual task.

Coding generally uses temperature 1.0, top-p 0.95 and 1M context; visual-agent evaluation uses a different setup including 512K. Network restrictions, removed Git histories and build/package-cache cleanup reduce leakage. Exploit-seeking behavior remains. These are author-reported results, not independent reproduction.

The report attributes post-training gains mainly to tasks, environments, verification and scale; it does not claim a new core RL algorithm or isolate every component. CED builds on YOCO, Single-Pass mixing on mHC, and memory-update balancing on earlier momentum/balancing methods. Architecture and engineering co-design are useful without implying that every ingredient is new.

Missing reproduction details include SFT counts, exact mixture/filter yields/masks, unique RL tasks and environments, accepted-task rate and construction cost, per-task CPU/RAM, node inventory, GPU-hours and calendar time. Complete reward coefficients, group sizes, staleness thresholds, domain schedules, merge rules and teacher settings also remain unspecified. Millions of live containers are not millions of tasks; 45T pretraining tokens are not an RL budget.

Reference inference code is not the production serving stack. Its straightforward all-layer path and out-of-scope speculative loop do not reproduce production CED skipping, bounded replay or DSpark scheduling. Verify actual execution paths before benchmarking those claims. Serving Models → connects cache ownership and prefill/decode separation. DeepSeek V4.1 report ↗ §§5.3–6 · Model card ↗ · Reference implementation ↗

MiMo: scale exploration and feedback together

MiMo Pro is 1.02T/42B, 70 layers, sixty local and ten global; Flash is 310B/15B, 48 layers, thirty-nine local and nine global. Local windows are 128. The first global block has a dense FFN; subsequent experts have no always-active shared expert. Pro trains 27T text+3T omni-modal, Flash 26T+22T.

The 681M vision encoder combines twenty-four local/four global layers and alternates image serialization axes. Audio has 25Hz residual-quantized codes and 6.25Hz continuous patch features. Muown row-norm-controlled matrices and AdamW embedding/head/router paths support pretraining; QAT prepares MXFP4 rollouts.

RL couples rollout compute, environment breadth and grader compute. At 1,568 prompts×16=25088 trajectories, 2.7–3.7B training tokens imply roughly 107.6–147.5K tokens per trajectory. A full step is not one rollout.

The stated task mixture is coding 68%, general tools 12%, visual 13%, context following 3%, cyber 4%. Pro's reported $2.6M RL cost splits rollout/training/grading 43.8/43.5/12.7%; Flash is $0.9M. These are run-specific economics, not current prices or a universal scaling law.

GRS builds reusable task-specific solution/behavior rubrics; $R=R_{\rm test}S_{\rm solution}S_{\rm behavior}$ differentiates passing solutions while keeping failures zero. GAR compares passing trajectories, rejects confirmed hacks and redistributes existing positive advantage with $\lambda=\frac{\sum_{i\in P} A_i}{\sum_{i\in P} f_iA_i}$. Here P contains eligible passing trajectories with positive advantage, and f is their quality factor. The denominator must be positive. GAR cannot create signal from a zero-advantage all-pass group by mere multiplication.

Four-harness training reportedly raises held-out-harness mean pass rate about 50→66%. This is interface generalization evidence, not a claim that every harness should always receive equal weight. MiMo-V2.6-Pro source ↗

MiMo: first create contrast, then allocate credit

More samples do not guarantee more feedback. If sixteen attempts all pass a binary test, the group-relative signal can vanish. MiMo adds two different grader paths. GRS can create reward differences among passing attempts. GAR allocates existing positive advantage in groups that already have contrast.

Groupwise Reward Synthesis jointly examines several offline attempts, the task and repository. Solution rubrics cover correctness, edge cases and codebase fit. Behavior rubrics cover evidence gathering and verification discipline. Individual online trajectories reuse those criteria; the grader need not rebuild them for every sample.

Groupwise Advantage Redistribution instead reads the current group’s specifications, patches, repository state and test outputs. It ranks suitable, precise, minimal changes while checking side effects and craftsmanship. Confirmed hacks receive zero before group statistics are recomputed.

The common factor preserves eligible positive mass before any additional cap. A final mean subtraction restores zero group mean. Multiplying zero advantages cannot rescue an all-pass binary group. A cap or broader soft-reward eligibility changes the conservation argument. Agent grading → includes the interactive allocation model.

The reported GAR ablation restrains uncontrolled turn and token growth while pass rate improves. Its groups and trajectory lengths change during training. This is useful evidence under the stated setup, not a perfectly compute-matched isolation of every component.

Response normalization prevents extra length alone from increasing a trajectory’s share. Local behavioral regularization addresses malformed calls, unavailable tools, garbled text and repetition. Positive flagged spans lose credit to clean spans; negative flagged spans receive more blame. Infrastructure failure is a separate exclusion, not evidence that a model action was incorrect.

Four sign-specific lower/upper ratio bounds begin with [0.2,5.0] and adapt to entropy. Low entropy permits more positive movement and constrains negative movement; high entropy does the converse. This feedback changes the update region, not the test verifier. Reinforcement Learning → owns the related policy-update controls. MiMo V2.6 report ↗ §§4–5.

MiMo: keep the update faithful to the rollout

The base update is GRPO-style, not a new wholly unrelated RL algorithm. Per-sequence token means precede the prompt/group mean, preventing long trajectories from gaining weight merely through token count. Four sign-specific ratio bounds adapt to entropy rather than one symmetric trust interval.

Local behavioral masks handle malformed calls, unavailable tools, repetition and garbled outputs. Positive flagged tokens lose credit redistributed to clean tokens; negative flagged tokens receive more blame. Infrastructure failures can be excluded. These controls preserve sign-specific mass under their stated redistribution rule.

R3 replays actual MoE choices. The exact top-p candidate set is replayed before probability renormalization. MXFP4 quantize/dequantize after each update aligns model arithmetic. Matching weights alone does not ensure that recomputed log probabilities describe the sampled route/support.

A trainable router collapses within twenty steps: load CV 0.78→2.0, peak load 6×→16× mean, cold experts 0.5%→22%. Restoring the router recovers balance without losing benchmark quality; freezing it stabilizes this run. That intervention is stronger evidence than a correlation, but not a theorem that all RL routers must be frozen.

DFlash is a five-layer dense sliding-window block drafter with bidirectional draft interactions and up to 1,024 previous backbone positions. Six drafts are the RL default. The reported FP8 throughput gain of about 10.3% is workload-specific; acceptance length alone does not select the best block size. MiMo-V2.6-Pro source ↗

MiMo: protect mixture and reward integrity

Sample→Sequence→Context→Segment records branches, compaction, masking and cache reuse. An external Agent Loop owns tools/environment/grading; inference remains token-in/token-out. Metadata scheduling is separate from bulk tokens, images, routes and probability records in distributed storage.

Across twenty-five sources, mean generated tokens vary 90× and active duration 66×. Adaptive concurrency, deficit-aware dispatch, predicted KV demand and limited replay protect the intended mixture. A fixed concurrency can let fast sources dominate. This is data mixing under latency, not merely increasing throughput.

Environment cleanup, network isolation, hacking-agent probes, offline audits and online hack rejection form a defense chain. Confirmed hacks below 2% are reported under that detection process, not proof that undetected exploits are absent.

Thirty Pro steps take 123.1h and Flash 81.8h. The failure list includes memory hardware errors, cyber-cluster failure, unreachable graders, stale restart length estimates, extreme expert load and host packing OOM. Recovery changes the sample and resource distribution if its state is incomplete.

MOPD2 uses whole student trajectories where rewards support them and prefix-conditioned next-turn student continuations elsewhere. SFT demonstrations become contexts, not fixed response targets. Domain teachers score student-generated mistakes without paying to regenerate every prior turn.

The open 9B student uses 77.4B distilled tokens, 27.2B with SFT loss, plus domain GRPO. SWE Verified 61.1→66.2, Terminal 37.1→52.8, Cyber mini 31.3→47.0 and Visual Coding mini 64.0→72.4 support transfer below frontier scale, not isolation of every component. Roughly 7K released environments make parts inspectable. MiMo-V2.6-Pro source ↗

MiMo: keep failure, transfer and cost evidence together

Pro and Flash use hidden widths 6,144 and 4,096 respectively. The first global block uses a dense FFN; subsequent sparse FFNs have no shared expert. The 128-position local window bounds those layers’ reachable KV exchange during context parallelism, not total communication or periodic global-attention work.

The 681M vision encoder alternates row-major and column-major serialization so narrow windows connect both image axes across layers. Audio uses discrete residual-quantized codes at 25 Hz and continuous patch features at 6.25 Hz. The different rates describe different representations, not a single interchangeable stream.

Pretraining broadens knowledge, perception, tool conventions and failure recovery before RL exploration. Context grows 32K→256K, then agent mid-training extends to 1M. Muown controls matrix row norms; embeddings, head and router use AdamW. QAT prepares MXFP4 experts for rollout arithmetic. More initial competence changes what the policy can discover, without establishing that the disclosed token mixture is optimal.

Tasks include real issue/PR repairs, workflow/specification/source-code synthesis and longer horizons. Deterministic local files and mocked software expose API, CLI, GUI or MCP interfaces. OSS-Fuzz reproductions use sanitizer evidence. Task, environment and verifier form one object; a plausible instruction with an unreliable checker supplies unreliable learning. Agent environments → keeps the detailed task mixture and harness-transfer context.

R3 replays expert indices; top-p replay restores the exact candidate set before probability renormalization. Quantize/dequantize after each update aligns MXFP4 arithmetic. These controls address route, support and arithmetic mismatch separately. None turns an arbitrarily stale trajectory into a fresh on-policy sample.

With a trainable router, load CV rises 0.78→2.0, peak load 6×→16× and cold experts 0.5%→22% within twenty steps. Restoring only the router restores balance without losing benchmark quality. Freezing it stabilizes this run. That intervention supports causality here, not a universal prohibition on router learning. MoE routing → holds the mechanism and replay context.

Thirty Pro steps require 123.1 hours; Flash requires 81.8 hours. Those are run durations, not per-trajectory latency or a compute-matched architecture comparison. Restart recovery must restore mixer deficits, length estimates and sample provenance. Across 25 sources, 90× generated-token and 66× active-duration variation makes fixed concurrency a biased data mixer. Adaptive concurrency, deficit scheduling, predicted KV demand and limited recovery replay preserve the intended distribution.

Sample→Sequence→Context→Segment separates branches, compacted contexts and trainable spans. The external Agent Loop owns environment setup, tools, grading and cleanup. The driver schedules metadata; bulk tokens, log probabilities, routes, top-p sets and images are stored once for their consumers. Driver memory otherwise becomes the batch ceiling. Trajectory lifecycle → explains the state boundaries.

Reward defense combines correct-use mid-training, cleanup of logs, patches, caches, bytecode and post-base Git references, network isolation, iterative hacking-agent probes, offline suspicious-success audits and online hack rejection. The reported confirmed-hack rate below 2% depends on that detection process. It does not bound undetected attacks.

MOPD2 uses full student trajectories where rewards support them. Elsewhere, Teacher-Prefix and SFT-Prefix histories supply assistant decision points. The student generates only the next turn; domain teachers score that student continuation. Demonstrations become contexts, not fixed target answers. This avoids replaying all earlier turns and exposes teachers to student mistakes in game development, research and embodied tasks with expensive or unavailable terminal verification. Distillation → supplies the general student-prefix mechanism.

MiMo-V2.6-Distill-Qwen-9B uses 77.4B distilled tokens, 27.2B contributing to SFT loss. The report says domain GRPO improves all eleven reported evaluations; these four examples do not stand in for an independently checked full benchmark census. Roughly 7K released environments, composable mini-harnesses, verifiers and an end-to-end RL framework permit smaller-scale inspection. The result supports transfer below trillion-parameter scale, not isolation of every frontier component.

DFlash’s five-layer dense windowed drafter predicts a future block from backbone anchor features. Draft interactions are bidirectional; context access is up to 1,024 preceding backbone positions. Six drafts are the RL default. After adaptation on early rollouts, reported FP8 per-node throughput improves about 10.3% on that long-context workload. End-to-end throughput, not accepted length alone, selects the useful block size. Speculative Decoding → explains target verification.

The report supports sustained mixed-domain gains, held-out-harness transfer, the router intervention and smaller distilled experiments. It does not isolate every combined technique. Frontier comparisons retain harness/evaluation differences; learned graders add bias; disclosed dollars are report-specific economics. Iterative capability gains do not establish autonomous recursive self-improvement in the strong sense. Useful scale depends jointly on exploration breadth, feedback resolution, throughput and stable updates—not a dimensionally valid multiplication of four measured quantities. MiMo V2.6 report ↗ §§6–11 · Official model repository ↗.

Qwen3.8-Flash-Next: connect architecture, optimization and evaluation

The 26 August 2026 report studies a 125B-parameter MoE backbone with 6B parameters active per token, plus 51B n-gram embedding parameters outside the accelerator. Qwen presents it as a preview of the architecture direction for Qwen4. The report's technical emphasis is architecture selection, optimization and stability, not a full account of final agent post-training.

The mechanisms have separate owners: GDN recurrence →, QSA conversion →, Gated Residual →, N-gram allocation →, Muon groups →, Hyperparameter refitting →, Stability tests → and Distributed execution →.

What is actually disclosed about pretraining?

Main pretraining first uses the GDN/full-attention hybrid. At 256K continued-pretraining context, the indexer learns from dense attention for about 2B tokens, followed by about 200B tokens of joint sparse adaptation. These are conversion exposures, not the whole model's training-token total.

The report gives controlled proxy runs at 25B-A3B and 35B-A3B, a 4T-token small-model batch study, and a 419B-token larger-model LR study. Do not add those research budgets to the shipped checkpoint's exposure. It also compares the first 276B production tokens under matched conditions. No complete production corpus mixture or absolute full-run token budget is disclosed here.

The reported stability margin supports larger learning rates and batches. Those settings affect convergence and parallel throughput, while sparse attention and recurrent state address long-context execution. The final quality is evidence for the combined recipe; it does not isolate one universal cause.

Qwen3.8-Next architecture report, Abstract and §§1–3 (26 August 2026) ↗.

Qwen base evaluation: preserve the denominator of every efficiency claim

The new base wins all 14 comparisons against the 27B dense base and eight against the 397B-A17B predecessor. The largest remaining deficit is 2.59 points on MultiPL-E. The reported one-ninth training FLOPs combines roughly one-third active parameters and one-third training tokens. It is not a controlled ninefold gain from QSA, GR or Muon alone.

The evaluations use few-shot conditions: MMLU/Redux/MMMLU/INCLUDE use five shots; MMLU-Pro, SuperGPQA and GPQA use five-shot CoT; BBH uses three-shot CoT; GSM8K/MATH use four-shot CoT; MGSM uses eight-shot CoT. Code tasks use zero shots. EvalPlus averages HumanEval, MBPP and their plus variants. MultiPL-E covers eight listed languages. SWEBench-Pretrain is a pretraining variant, not SWE-bench Verified agent performance.

No complete end-to-end serving benchmark or production wall-clock bill accompanies the one-ninth FLOP claim. Small benchmark differences have no general statistical significance without uncertainty estimates. A 1M-token retrieval evaluation also does not certify reliable 1M-token agent behavior.

What the report does not reveal

Attention-distribution distillation in QSA is documented. It is not evidence of an undisclosed teacher-student agent-distillation pipeline. The repository's generic advice that frameworks can fine-tune with DPO or GRPO does not identify the algorithm used to train this checkpoint.

Finally, report two late-stage failures alongside the successes: removing positional encoding raises endless-generation rates, and sparsifying GR reads harms post-training. Better pretraining loss is not a complete release criterion. Failure definitions and stress results → · MTP measurements →.

Qwen3.8-Next architecture report, §4, Table 11 and disclosure scope across the full report (26 August 2026) ↗.

Read each evaluation with its harness

Record dataset version, pass@k versus pass@1, resolved versus judge score, tools, effort, context, sampling and retries. DeepSeek ZeroBench 49.0 is pass@5. Its report NL2Repo 65.4 differs from the checked model-card 64.0; do not silently merge the two artifacts.

Internal/custom tasks, repaired subsets, best-of-harness selection and model judges limit reproducibility. A competitive complete system does not identify the causal contribution of each architectural novelty. A failed or omitted ablation remains a gap, not negative proof.

Extract a testable change, not a copied recipe

Choose one bottleneck: cache growth, indexer cost, routing imbalance, weak task coverage, poor verifier resolution or rollout stragglers. Keep the baseline, data, resource denominator and deployment contract fixed enough to answer that question.

For a sequence mechanism, test masks, selected positions and intermediate state against a small exact reference. For a training change, run matched proxies and scale confirmation. For agent learning, audit executable tasks and verify that the intended mixture actually enters updates.

Retain the surrounding dependencies. Index sharing requires adaptation; QAT requires a matching export; partial rollouts require behavior-policy records; learned graders require adversarial checks. A technique detached from those conditions is not the same experiment.

Run Design →, Precision → and Parallelism → own the reusable procedures. Complete source reports remain available for details not independently reproduced.

References and full report context

Llama 3 405B ↗ · Qwen2.5 72B ↗ · MAI-Base-1 / MAI-Thinking-1 ↗ · GLM-5 ↗ · GLM-5.3-Flash ↗ · Kimi K3 ↗ · DeepSeek-V4.1-Flash ↗ · MiMo-V2.6-Pro ↗

GLM-5 configuration ↗ · GLM-5.3-Flash configuration ↗ · Kimi release explanation ↗

Stage playback is explanatory. Uniform two-byte inventory and token/step arithmetic are calculations, not measured serving memory, run speed or capability predictions.