Optimization

A gradient is not yet an update

For local quadratic curvature H, the estimated loss change is $g^{\mathsf T}\Delta\theta+\tfrac12\Delta\theta^{\mathsf T}H\Delta\theta$. Descending the linear term is not enough if the quadratic term overwhelms it. On a positive quadratic eigenvalue λ, plain gradient descent requires 0<ηλ<2 for convergence along that direction.

Neither this local calculation nor a decreasing training loss proves good generalization. For the gradient and curvature geometry, see Differentiation and Optimization Geometry →. The loss definition → determines what improvement means.

Remember directions, not just the current batch

An exponential moving average (EMA) gives recent gradients greater weight. Consistent directions accumulate; alternating directions cancel. After a direction changes, momentum initially continues to include the earlier direction.

The approximate memory is $\frac{1}{1-\beta}$ steps when β is near one. The exact e-folding time is $-\frac{1}{\log\beta}$ for $0<\beta<1$. At $\beta=0.9$, the approximation is ten steps; at $\beta=0.99$, one hundred. These are steps, not tokens: changing batch size changes the amount of data in that memory.

Momentum is not merely an independent learning-rate multiplier. In a nonstationary sequence it changes direction, response delay and correlations as well as scale. Nesterov variants read a different combination of current gradient and momentum; identify the implementation convention.

Watch a direction reverse

Adapt coordinates with two moments

The numerator smooths direction. The denominator rescales coordinates according to recent magnitude. A persistently large coordinate gradient can receive a comparable normalized step to a smaller one. Adam still has one base learning rate, but effective coordinate updates differ.

v is not a covariance matrix or a Hessian estimate. Even $v-m^{\odot2}$ is only a variance-like diagnostic under compatible averaging/stationarity conditions; the two buffers usually use different β values. Calling Adam a second-order optimizer obscures this distinction.

Correct the startup, not all uncertainty

For a constant gradient g, the correction recovers $\widehat m=g$ and $\widehat v=g^{\odot2}$ at every step. It does not make a noisy mini-batch equal to the full-data gradient, nor does it make the ratio unbiased.

When ε is negligible and g≠0, the first adaptive step is approximately −η sign(g), not proportional to |g|. This explains the sign-like intuition but does not make every later Adam step SignSGD. Momentum, changing magnitudes and ε matter.

ε affects very small-gradient coordinates. With g² far below ε², the denominator is ε-dominated; the sign approximation fails. Epsilon location also matters: $\sqrt v+\varepsilon$ and $\sqrt{v+\varepsilon}$ are different rules.

Keep shrinkage outside the moments

Adding λθ to the loss gradient causes Adam to accumulate and adapt that term. It is not the same as applying a uniform shrink factor outside the moments. For ordinary SGD without momentum, the two forms coincide under matching conventions.

For $\theta=10$, $\eta=0.01$ and $\lambda=0.1$, decay subtracts 0.01 before considering the adaptive contribution. A constant gradient of 2 gives an adaptive subtraction near 0.01 at step one. These are separate effects; decay is not evidence that the training objective decreased.

Implementation order, skipped parameters and mixed precision matter. For example, a parameter with no gradient may be skipped rather than decayed. Use the actual optimizer rule, not a generic “Adam plus L2” label. AdamW algorithm ↗

Weight decay discounts update history

For 0<ηλ≪1, an older contribution decays approximately as $e^{-\eta\lambda k}$. Its e-folding timescale is about $\frac{1}{\eta\lambda}$ steps. With $\eta=10^{-4}$ and $\lambda=0.1$, that is about 100K steps.

With a varying schedule, replace powers by products of $(1-\eta_t\lambda)$. Equal ηλ gives equal direct forgetting timescale, not equal training dynamics: changing η also changes the trajectory and future u. This is a useful interpretation of decay, not a proof that two recipes are interchangeable. Learning rate and decay as memory ↗

Read settings as a recipe, not a law

These are candidate settings from the atlas sources, not package defaults or current universal recommendations. AdamW is a useful controlled baseline. Smaller β₂ reacts faster to scale changes but adds estimator variability. Higher β₁ smooths more and responds more slowly.

Why treat a matrix differently?

A hidden weight W is a linear map, not only a flattened parameter vector. Adam rescales entries separately. Elementwise SignSGD discards each entry’s magnitude. Muon instead normalizes the singular geometry of a momentum matrix.

The rectangular partial polar factor is often called Muon’s “matrix sign.” This is not the analytic matrix-sign function based on eigenvalues of a square matrix. A rank-deficient matrix uses only nonzero singular directions.

Equivalently, $\operatorname{polar}(M)=M\left[(M^{\mathsf T}M)^{1/2}\right]^\dagger$, where the dagger is the Moore–Penrose pseudoinverse. A diagonal matrix reduces to coordinate signs. A column vector instead becomes L2-normalized: treating a vector as a diagonal matrix versus a column gives different geometry.

A conceptual update is $W\leftarrow(1-\eta\lambda)W-\eta\operatorname{polar}(M)$, with optional shape scaling. Actual Muon implementations differ in momentum normalization, Nesterov combination, polynomial approximation and scale. Do not identify an ideal formula with every implementation.

The norm defines steepest descent

A spectral-norm budget limits the largest linear action of the update on a unit input. A Frobenius budget limits total flattened energy. They define different steepest directions.

For the quadratic penalty $\frac{\|\Delta W\|^2}{2\eta}+\langle G,\Delta W\rangle$, the Frobenius solution is −ηG. With spectral norm, one solution is $-\eta\|G\|_*\operatorname{polar}(G)$. The nuclear-norm multiplier matters: the constrained direction and the penalized update are not identical formulas without rescaling.

This explains the appeal of matrix-aware geometry, not why it must outperform Adam on every architecture. The momentum direction can also differ from the current gradient. Learning rates are meaningful only together with normalization and scaling conventions.

Approximate the polar direction without an SVD

Computing a full singular-value decomposition (SVD) per matrix per step is expensive. Newton–Schulz-style polynomial iterations use matrix multiplications instead. Start with $X_0=\frac{M}{\|M\|_F}$, with an explicit zero-matrix case.

The Taylor-derived coefficients (15/8,−5/4,3/8) move normalized nonzero singular values toward one. Tiny singular values can need more iterations. Zero stays zero; finite precision can change effective rank.

The checked Muon implementation instead uses (3.4445,−4.7750,2.0315), BF16 and a small fixed iteration count. Its own explanation says it does not converge to exactly UVᵀ everywhere. This fast approximation is not the same as the ideal polar factor. Muon implementation ↗

Use XXᵀ instead of XᵀX when the row dimension is smaller; the corresponding algebraic multiplication order is equivalent. Transposing rectangular matrices reduces intermediate cost. Test accuracy and dtype on the actual shapes rather than assuming more iterations are always better.

Transform singular magnitudes step by step

Normalize shape-dependent update RMS

A full-rank 4×4 polar update has RMS 0.50; a 4×16 update has 0.25. Equal learning rates do not give equal entry-wise RMS.

The original Moonlight-style rule multiplies by $0.2\sqrt{\max(n,m)}$, targeting RMS≈0.2 for full-rank ideal factors. Rank deficiency and approximate singular values change that equality. The 0.2 is an empirical calibration, not a mathematical constant.

The checked Keller Jordan code uses a different aspect-ratio factor, $\sqrt{\max(1,n/m)}$, with its own learning-rate convention. Do not combine the two scaling recipes blindly. Track update RMS and update/weight ratios for each parameter group.

Choose which geometry is normalized together

A combined Q/K/V projection joins multiple heads into one stored matrix. Whole-matrix orthogonalization couples the heads through shared singular directions. Splitting by functional head changes that coupling.

Kimi K3’s original note describes per-head Muon; GLM-5 describes Muon Split for MLA Q/K/V up-projections. Each head’s momentum block is processed separately, then concatenated. This is not equivalent to a different scalar learning rate per head.

The reported GLM comparison matches a GQA-8 baseline while keeping attention logits stable without clipping in that setting. Smaller polynomial matrices can reduce work, but partitioning changes update geometry as well as cost. Verify head axes, layouts and grouped-query sharing rather than splitting an arbitrary storage dimension. Kimi K3 report ↗

Assign parameters by function

Two groups must not update the same tied parameter twice. A shape test alone is insufficient: embeddings are matrices but serve a different function from hidden projections. Check the parameter identity, not only its name.

Muon typically stores momentum without Adam’s second-moment buffer for those matrices, but total memory includes gradients, master weights, temporary products and distributed state. Communication and sharding can offset arithmetic savings. See Distributed Training →.

Weight spectra are evidence, not an explanation by themselves

A higher normalized singular-value entropy means energy is spread more evenly. Reported Moonlight-style analyses find more uniform spectra with Muon. The hypothesis is better use of matrix capacity and less compressibility; entropy alone does not prove either downstream benefit.

Normalize the same spectrum definition, matrix role and rank threshold when comparing optimizers. An optimizer’s update spectrum is not the trained weight spectrum. Also test pretraining-optimizer × fine-tuning-optimizer combinations: switching an Adam-pretrained model to Muon for SFT need not reproduce Muon pretraining gains.

What a larger batch actually changes

A larger B reduces estimator covariance in this model. Correlated examples, repeated tokens and changing data mixtures reduce the effective independent sample count. Larger batches do not guarantee better generalization or better progress per compute.

Multiplying signal and noise by η leaves their instantaneous ratio unchanged. For $\operatorname{SNR}=\frac{\|g\|_2^2}{\operatorname{tr}(\Sigma)/B}$, η cancels. A smaller learning rate reduces absolute step noise and curvature effects; it does not improve this estimator’s SNR by itself.

When the mean gradient shrinks while sampling variance stays substantial, SNR can fall. Learning-rate decay may help dynamics then, but it does not directly reduce gradient-estimation variance. Increasing B does under the stated sampling assumptions.

Separate signal, step noise and data budget

Noise can influence generalization and exploration, but “small dataset means small batch is best” and “one data pass means large batch is best” are not laws. Data quality, objective, repeated examples, architecture and update count still matter.

Linear and square-root rules preserve different quantities

With frozen independent SGD gradients, $\eta\propto\sqrt B$ holds one-step update covariance $\frac{\eta^2\Sigma}{B}$ constant. It does not hold signal-to-noise ratio constant: batch already changes that ratio.

At a fixed number D of examples, there are D/B updates. Cumulative mean displacement is approximately $-\frac{D\eta g}{B}$; cumulative covariance $\frac{D\eta^2\Sigma}{B^2}$. Linear $\eta\propto B$ preserves both expressions in this frozen-gradient approximation. Real gradients and curvature invalidate unrestricted extrapolation.

The original AdamW example increases $B=256\to1024$ and $\eta=10^{-4}\to2\times10^{-4}$ using a square-root candidate. It is a candidate for a sweep, not a proved general Adam rule. Adaptive moments change the derivation; changing B also changes EMA memory measured in tokens, ε dominance, schedule duration and decay per token.

Record whether B counts examples, sequences, valid tokens or weighted loss elements. Gradient accumulation approximates a larger mean batch only when loss normalization and weighting match. BatchNorm statistics, randomness, clipping each microbatch and sequence-dependent weighting can break equivalence.

Why an adaptive optimum can turn over

A simplified Adam analysis replaces the update by coordinate signs. For Gaussian coordinate noise, the exact mean sign is $\operatorname{erf}\!\left(\frac{g_i\sqrt B}{\sqrt2\,\sigma_i}\right)$. The original blog uses the approximation below:

Assuming a shared noise/signal ratio κ², define the signal attenuation b below. A Hessian-based local approximation then gives:

The expression peaks at $b=b_{\mathrm{noise}}$ only if the relevant denominators are positive and the optimum lies in the reachable $b\in(0,1]$ range. Negative/zero cross terms, unequal noise ratios or indefinite curvature invalidate the simplified interpretation. $b_{\mathrm{noise}}$ is not Adam’s $\beta_1$ or $\beta_2$.

This model motivates a possible learning-rate “surge”: growth with B can flatten or reverse. It does not prove noise always corrects sign updates or that every huge batch needs lower LR. The original linked analyses remain references; the 10542 page was unavailable for a fresh full review in this pass.

Search a joint near-optimal region

Hold architecture, data mixture, tokenizer, precision, loss reduction and schedule family fixed. Choose feasible global token batches from memory and throughput measurements. At each batch, sweep peak LR on a logarithmic grid.

Use equal token budgets for data-efficiency comparisons and also report elapsed time. Keep enough horizon to expose moment-state and routing instability; a brief favorable loss slope is not a stable long run. Repeat promising settings with more than one seed and a held-out evaluation.

Map the region within a stated tolerance of best validation loss. Prefer a robust region with good throughput and clipping behavior over one noisy winning point. Test larger-scale checkpoints before extrapolating.

Mean gradients of independent microbatches can estimate trace covariance: B_micro times their sample variance estimates per-sample trace under matching assumptions. Repeatedly evaluate at fixed checkpoint and record the sampling definition. This estimates a local noise scale, not a permanent global optimum.

Control early functional changes

Linear warmup uses $\eta_t=\eta_{\mathrm{peak}}\frac{t}{T_w}$ for $0\le t\le T_w$. Here, t counts steps from the start, $T_w>0$ is the warmup duration, and $\eta_{\mathrm{peak}}$ is the peak learning rate. Warmup limits early parameter changes while activations, optimizer statistics and the data-driven direction settle. Large initial gradients are one possible issue, not a necessary property of random initialization.

Bias correction already removes zero-start EMA attenuation under stationarity. Warmup has a different role: noisy ratios, rapidly changing states, normalization, depth and a large chosen peak can still destabilize updates. It is not waiting for Adam to learn a Hessian.

The original heuristics suggest 1–5% of pretraining steps, or 50–500 steps / 0.1–1% for some short fine-tuning runs. Treat them as examples, not guaranteed industry rules. Larger batches do not automatically make mean gradients larger or require longer warmup.

Too little warmup can expose an unstable peak early. Too much can waste budget at small steps; it does not prove the model is trapped in a bad local minimum. Sweep duration jointly with the peak and inspect functional update scale.

The original Transformer schedule grows linearly, then decays with the inverse square root of step:

Changing model width and warmup also changes its peak; do not compare schedules by shape alone.

Warm up, remain stable, then decay

Warmup–Stable–Decay (WSD) separates three decisions: reach a peak, keep it during an extendable plateau, then begin a cooldown for a selected endpoint. Cosine normally begins decreasing after warmup and ties the trajectory to a chosen horizon.

The practical advantage of WSD is shared trunk training. Several token-budget comparisons can branch from plateau checkpoints and receive their own cooldown. Comparing a cooled checkpoint with an uncooled plateau is not a fair final-quality comparison.

Late decay may refine the endpoint, but a sharp loss decrease is not proof that all useful learning happens in the cooldown. If data changes at the same time, schedule and mixture effects are confounded.

Discrete drops around 80% and 90% of tokens are one multi-step example. Linear, cosine or other cooldowns are possible. Extending training after a cooldown is not the same as simply extending the plateau; document resume policy and optimizer state.

Compare schedule shape at fixed peak

Make small experiments predictive

Empirical hyperparameter laws fit settings across compute budgets. Using $C\approx6ND$ for a dense training approximation, the retained empirical fits are $\eta_{\mathrm{opt}}=0.3118\,C^{-0.1250}$ and $B_{\mathrm{opt}}=0.2920\,C^{0.3271}$. Their coefficients require the original compute/batch units and setup; do not insert arbitrary GPU-hours or token units.

These fitted trends are priors, not universal laws. Different language/domain mixtures, data quality, tokenization, objectives, precision and architecture can move the region. Fit locally over several budgets, record residuals and uncertainty, and revalidate after distribution changes.

Maximal update parametrization (μP) addresses a different axis: width. Stable initialization variance alone does not ensure stable update-induced features. For $y=xW$, inspect $\Delta y=x\Delta W$, not merely $\|\Delta W\|$. Forward signals, backward sensitivities and functional updates need compatible scaling.

A width-stable base learning rate is not a width-stable raw learning rate in every tensor. The reference μP implementation divides hidden-weight Adam LR by the width multiplier and treats readout/input dimensions separately. This corrects the old blanket hidden-matrix LR statement. μP parameter-group scaling ↗

Define base and changed-width shapes, configure initialization/readout/optimizer together, run coordinate checks across widths, tune on a small member and validate larger members. The reference family keeps depth fixed; changing depth is not automatically covered. Schedulers must preserve per-group ratios rather than overwrite every LR with the same absolute value. A resumed model must not be reinitialized by shape registration.

μP transfer and compute-law fitting are complementary. Neither eliminates the need to test changed heads, sparse experts or new data. The functional-scaling explanation below connects the spectral view to these parameter-group rules.

Clip gradients before the optimizer transforms them

Value clipping clamps each coordinate and can change direction. Global clipping’s SGD update satisfies $\|\Delta\theta\|_2\le\eta\tau$, but Adam transforms the clipped gradients through moments and decay. It does not give that same universal bound on the final Adam update.

Accumulate correctly weighted gradients, unscale mixed-precision gradients, check finiteness, compute the logical global norm and clip before the optimizer step. Sharded norms require the appropriate collective and must not double-count replicated copies. Clipping each microbatch is generally different from clipping the accumulated gradient.

Use the norm distribution and clipping rate to choose τ. A threshold of 1.0 or 0.5 is a trial setting, not a unit-independent safeguard. Frequent clipping can be intentional or indicate an overly low threshold, a reduction bug or pathological batches; it is not automatically wrong.

Inspect actual parameter changes as well as raw norm. Stable norms can coexist with poor generalization or adaptive update spikes. The diagnostic table below separates raw gradients from actual parameter changes.

Qwen Muon: choose the operator before orthogonalizing it

Muon normalizes a momentum matrix through a finite orthogonalization procedure. The unit of that operation should match a meaningful linear map, not whichever tensors the training framework stores together.

Qwen uses Nesterov momentum 0.95 and eight Newton–Schulz steps with the Polar Express coefficient schedule. More accurate orthogonalization reduces the frequency and magnitude of spikes in their stress tests. The pre-iteration Frobenius normalization uses a numerical constant of 10⁻¹⁴.

The report hypothesizes that router outputs lack the shared linear structure that makes orthogonalization helpful. That explanation is not established by the observed instability. Similarly, the shape explanation for GR is an author interpretation.

Split fused QKV and GDN input gradients into per-head semantic submatrices. Split SwiGLU fc1 into gate and up halves. Orthogonalize each eligible part independently, then restore the original layout. Otherwise singular directions couple unrelated operators and the shape scale uses the wrong dimensions.

# Schematic update grouping; not runnable Muon code.
for parameter in fused_parameters:
    for part in semantic_parts(parameter):
        if uses_muon(part):
            direction = polar_express(momentum(part), steps=8)
            direction *= 0.2 * sqrt(max(part.shape))
            apply_update(part, direction)
        else:
            apply_adam_family_update(part)
# Preserve one optimizer owner/state per logical parameter.

Splitting QKV/GDN improves both loss and benchmarks. Splitting fc1 leaves loss essentially unchanged while slightly improving benchmarks. These observations reinforce the distinction between storage fusion and update geometry.

The report does not supply every production learning rate, decay coefficient or polynomial coefficient. Eight steps alone do not reproduce Polar Express. Retain exact coefficients, normalization dtype, zero-matrix behavior and state ownership when implementing it. Distributed Muon and kernel overhead → · Retune LR and batch together →.

Qwen3.8-Next architecture report, §3.1, p. 16 (26 August 2026) ↗.

Model recipes use different groups and phases

These are disclosed settings retained from the original model notes, not recommendations to copy. Full data, architecture, precision and RL conditions remain in their report owners.

DeepSeek’s original note records Muon momentum .95, decay .1 and target update RMS .18; AdamW betas 0.9/0.95, $\varepsilon=10^{-20}$ and decay 0.1. Large conditional-memory tables get a 5× LR multiplier and 11 Sinkhorn-balancing iterations, $\tau=10^{-3}$ and $\varepsilon=10^{-20}$. Alternating row/column squared-update balancing is not Muon’s polar normalization and not mHC’s forward residual mixing.

45T / 100.6M gives roughly 447K updates only if the global token batch stays fixed. It is not GPU-hours. Context grows from 64K to 1M at 34T; the phase transition also changes workload. Distributed state ownership and optimizer memory are part of this recipe.

GLM’s DSA indexer warmup uses 5×10⁻³→2×10⁻⁴; sparse adaptation uses 10⁻⁵. Those are different parameter/stage objectives, not contradictions in one peak LR. MAI reports less aggressive decay improves post-RL results; that is an observation about later trainability, not a universal pretraining loss law.

MiMo’s GRPO-like prompt means and entropy-controlled four-way clipping act on policy ratios/credit assignment. They are not gradient-norm clipping or a new Muon polynomial. Keep RL aggregation and rollout weighting in the RL topic rather than describing every “optimization” mechanism as an optimizer update.

Measure the update that the model receives

Verify gradient reduction, accumulation weighting and parameter-group identity first. Then log raw/after-clip norms, moment statistics, update RMS, update/weight ratio and effective decay by group. Layerwise values often reveal what one global scalar hides.

For Muon, test the zero matrix, rectangular shapes, rank deficiency, singular-value behavior, head partition and chosen polynomial dtype. Check state ownership and communication. A short pilot must include resume from a full optimizer checkpoint.

Compare optimizers at matched unique tokens, FLOPs and elapsed-time constraints; report all three rather than claiming they are equivalent. Include delayed instability, downstream transfer and post-training behavior. Prefer a stable near-optimal region to a single early loss winner.

Batch, learning rate, moment memory, clipping, decay and schedule form one recipe. Change one axis at a time when testing a mechanism, then retune jointly for a performance comparison. Neither fair experiment should be confused with blindly copying every Adam setting into Muon.

References and connected topics

Adam ↗ · Decoupled weight decay ↗ · Muon implementation and conventions ↗ · μTransfer ↗ · Smol training playbook ↗

Batch/LR analysis ↗ · Epsilon and LR scaling ↗ · Batch/LR foundations ↗ · EMA and batch ↗ · Muon geometry ↗ · Muon experiments ↗ · Muon details ↗

Optimizer settings belong to the model, data and phase that produced them. Frontier training recipes → · RL objectives and rollout consistency → · Run stability →

The visual experiments use exact synthetic calculations, not measured model-training results.