Precision and Quantization

Start with the arithmetic contract

An “FP8 model” can retain FP32 optimizer states and sensitive operations. A four-bit cache can be dequantized before attention. A weight-only quantized model can still use higher-precision activations. Name the boundary before comparing speed, memory or accuracy.

W8A8 means eight-bit weights and activations under a stated scheme. It says nothing by itself about cache precision, accumulators, scale granularity or kernel fusion. Storage savings are not automatically arithmetic speedups.

Spend bits on range or resolution

A floating-point value has a sign, an exponent and a significand. More exponent bits extend range; more fraction bits add resolution within a range. FP16 offers more local resolution than BF16, while BF16 retains a much wider exponent range. Neither removes rounding error.

The suffix and exceptional-value convention matter. E4M3FN is not every format called E4M3. Hardware can flush subnormals or choose different overflow handling. The interactive model below explicitly chooses finite saturation.

E4M3 often suits forward operands; E5M2 trades resolution for gradient range. This is a recipe choice, not a law that every backward operand must use E5M2. Transformer Engine 2.13 format primer ↗

A tiny gradient can disappear before underflow

Range and resolution are different constraints. FP32 can represent a positive value such as 10⁻⁸ near zero, yet adding it to 1 rounds back to 1. The value has not underflowed; the sum cannot retain its contribution at that magnitude.

This distinction matters in softmax. Subtracting the maximum logit avoids exponential overflow, but a dominant class can still acquire a rounded probability of exactly 1. A backward path that then subtracts its hard target can lose the small negative gradient while other entries remain positive.

The right question is not only “which dtype?” but also “which intermediate expression?” A cancellation-aware rewrite can preserve the same mathematical objective at the same precision. See the precision-cliff demonstration and corrected cross-entropy. The broader format and kernel contract still applies: an upcast reduction, a final output cast and hardware subnormal handling are separate boundaries.

Move the grid with a scale

A format's exponent range is not enough for arbitrary tensors. Choose s to place useful magnitudes within the format. For finite format maximum $F$ and tensor maximum $a=\max_i|x_i|$, the simple no-margin choice is $s=\frac{a}{F}$. A zero tensor needs a defined safe scale, not division by zero.

Some APIs store the inverse scale $\alpha=\frac{F}{a}$ and cast $\alpha x$. Their “larger scale” means the opposite of this note's s. Record the convention in checkpoint and kernel metadata.

Choosing a smaller range improves resolution but clips extremes. Choosing a larger range avoids those clips but can round useful small values to zero. Percentile or learned ranges change the objective; max scaling is not universally optimal.

Explore actual floating-point bins

One outlier can move a whole group

If one tensor shares s, its largest magnitude can determine everybody's grid. A single large activation can therefore reduce the resolution of many ordinary values. The error is local to the scale group, not necessarily to the whole layer.

Per-token rows can isolate activation outliers between tokens. Per-feature columns can isolate them between channels. Weight-channel conventions depend on the matrix layout and output-channel axis. “Per-channel” is incomplete unless that axis is named.

Smaller blocks reduce this coupling, at the cost of metadata, alignment and extra scaling work. The best grouping depends on distributions and the product being computed. More groups do not by themselves prove a faster or better model.

These are logical broadcast shapes, not packed kernel layouts. For W shaped [D,M], output channels are columns; a transposed storage convention changes that axis. Scale metadata and conversion work remain part of the cost.

Compare scale ownership

MXFP8 and NVFP4 add scale structure

MXFP8 groups 32 values under an E8M0 power-of-two scale. The grouping is part of the representation, not just an observer attached to ordinary per-tensor FP8. Group boundaries and hardware layout constrain implementation. MXFP8 documentation ↗

NVFP4 combines E2M1 values with block scales in E4M3 and a tensor scale in FP32. The documented recipe uses 16-value activation/gradient groups and 16×16 weight blocks. That second scale level is important: E2M1 alone is not the complete representation. NVFP4 documentation ↗

The associated training recipe can use stochastic gradient rounding and a random Hadamard transform to spread outliers. For an orthogonal $H$, $(AH)(H^\top B)=AB$ before quantization. The transform cancels algebraically; quantization errors do not cancel automatically. Keep layout, seed and normalization consistent.

Do not rank FP4 as “only for inference.” Specific training recipes use it too. Equally, do not assume an arbitrary four-bit tensor can feed those kernels. Format, scale hierarchy, supported shapes and hardware form one contract.

The transpose is another quantization problem

A row-grouped tensor and its column-grouped transpose generally need different scales. Transposing already rounded data cannot recover the high-precision source needed for the other layout.

Derive both quantized views from the original higher-precision tensor when required by the recipe. Otherwise a second quantization compounds error. Scale swizzling, padding and packed alignment are physical layout details that must match the kernel.

Measure the cost of producing and retaining both views. A theoretical low-bit GEMM advantage can disappear if conversion, transposition and metadata dominate the workload.

Choose when the scale is measured

Current scaling measures the tensor's present maximum before its cast. It can react to a new spike, but needs the observation and conversion work to be scheduled. Delayed scaling uses preceding maxima so observation can accompany the current cast.

A history maximum remembers large values for a chosen window. It reduces repeated clipping after a spike, but can leave a coarse grid after magnitudes return to normal. A latest-value policy reacts faster and forgets faster. Neither predicts an unseen spike.

History length 1,024 appears in specific recipes; it is not a mathematical constant. Include margin, update frequency, reduction ownership and history policy in the configuration. Current scaling is an available strategy, not universally “infeasible.” Delayed scaling documentation ↗

Watch a delayed scale miss a spike

Three matrix products need not share precision

Forward values, activation gradients and weight gradients have different distributions and sensitivities. Each product can use a different operand format while accumulating in a higher precision.

Evaluate a recipe at the product level. Reusing a cast, recomputing it, and keeping a transposed view have different memory and arithmetic costs. Optimizer state is a separate part of the training budget.

Accumulate accurately after rounding inputs

A higher-precision accumulator protects the sum of the represented products. It cannot reconstruct values discarded before multiplication. Conversely, precise operands do not protect a long low-precision sum.

For the exact attention example below, rounding probabilities [0.6, 0.3, 0.1] to [0.5, 0.25, 0] changes their product with [0, 2, 10] from 1.6 to 0.5. Perfect accumulation still gives 0.5. The lost input information is the issue.

Large reductions, cancellation and tiny updates need separate tests. The accumulator dtype in a kernel's name does not describe every intermediate or its final output cast.

Protect the sensitive path selectively

Normalization statistics, softmax reductions, losses and logits can be more sensitive than a large linear projection. The residual stream repeatedly transports error through depth. MoE routing can turn a small score change into a discrete expert change.

Keep a stable high-precision reference, then enable low precision in measured stages. Restore selected sensitive operations if evidence identifies them. Do not infer that all these operations must always use FP32 on every architecture.

Watch near-tied router logits, output-head sensitivity and small norm denominators. Preserve attention masks and stable log-sum-exp behavior. Scaling a tensor cannot fix a wrong reduction or a fully masked-row policy.

MoE routing → and Training Stability → supply the connected mechanisms.

Keep small updates and scaling state

A low-precision weight can fail to change when its update is smaller than the local bin. Higher-precision master weights let several small updates accumulate before the next cast. Optimizer moments often need their own precision budget.

Loss scaling multiplies the differentiated loss to keep gradients representable, then unscales them before the intended update. Tensor scaling maps an individual tensor into a format's range. They solve different problems; do not treat either as a universal stabilization knob.

Checkpoint histories, scale metadata, RNG streams for stochastic rounding and optimizer state. Activation recomputation must obey the chosen scale/RNG contract rather than accidentally consuming a new random stream or mutating history twice. A resumed run needs more than weights.

Run Design → covers full-state recovery.

Follow error through attention

Rounding can enter Q/K, the score product, softmax intermediates, stored probabilities, V, the PV product or the final output cast. A fused kernel can avoid storing full P and still have lower-precision intermediate boundaries.

Two kernels with “FP32 accumulation” can differ because they round operands or intermediates differently. Compare the complete path, including scale, layout and mask. A row with no valid keys needs a defined implementation policy; the ordinary sum-to-one statement does not settle it.

For small perturbations, score errors change P through the softmax Jacobian. Probability and value errors then give $\Delta O\approx\Delta P\,V+P\,\Delta V$. This approximation requires small perturbations; clipping and near-discrete routing are not small-error guarantees.

Define the expectation before calling error biased

For fixed inputs and deterministic rounding, the error is just a number. Calling it biased requires an expectation over a specified population. “Round to nearest” does not imply zero mean over arbitrary tensor values.

Nonnegative P and uneven V make probability errors consequential, but they do not prove one universal sign or magnitude. Correlation, row mass, cancellations and later nonlinearities all matter. Mean error alone can also hide damaging outliers.

The original Su Jianlin note motivates this diagnostic question. The following experiments are independent mathematical examples. They do not establish a measured bias for every fused kernel.

See probability rounding change the output

Unbiased rounding still has variance

Under these conditions, stochastic rounding is conditionally unbiased. It does not mean every sample is accurate. If independent probability entries are rounded, their sampled row can cease to sum to one, even when its expected mass is one.

For one output component, fixed exact values $V_j$ and independent probability-entry errors $e_j$ give $\operatorname{Var}(\Delta O)=\sum_j V_j^2\operatorname{Var}(e_j)$. Correlated errors add covariance terms. Clipping, approximate random thresholds or input-dependent scales can invalidate the simple conditional statement.

Renormalizing a rounded row is not a free fix: dividing by its random mass is nonlinear and can introduce another bias. Stochastic operand rounding does not prove an entire deep network or attention pipeline is unbiased.

A tiny perturbation can change a tie

At scale one, BF16 neighbors near one are 1 and $1+\frac{1}{128}$. The midpoint $1+\frac{1}{256}$ rounds to 1 under ties-to-even. Add $2^{-16}$ before the cast and it rounds to $1+\frac{1}{128}$ instead.

The perturbation is tiny, but crossing the midpoint flips a whole output bin. At a midpoint whose lower encoding is odd, ties-to-even instead chooses the upper neighbor. This worked example therefore does not establish a universal upward bias.

When a fused kernel changes a tiny intermediate term or reduction order, inspect boundary cases rather than concluding that a more precise sum must produce identical casts. This example is derived here; it is not attributed to the inaccessible source article.

Isolate casts before changing the model

Compute relative error with a defined denominator, such as $\max(\lVert O\rVert,\varepsilon)$ with $\varepsilon>0$. Report absolute error too, especially near a zero reference. Separate clipping frequency, underflow-to-zero and ordinary rounding.

Sweep sequence length, probability concentration, outliers, batch and representative inputs. A random short test does not certify long-context or real-data behavior. Inspect per-layer/per-head tails as well as averages.

A kernel switch or restored higher precision can localize an issue without proving its final cause. Validate forward outputs, gradients and end-to-end loss under the same input and reduction contract.

PTQ adapts the representation

Post-training quantization starts from an already trained checkpoint. It chooses scales, groupings, ranges and possibly reconstruction procedures without ordinary model-training updates. Calibration data helps fit input-dependent activation ranges.

PTQ is a useful baseline because it avoids a new training run. It can fail when outliers, channel range differences or very low bit widths distort a sensitive model. Its quality must be measured at the intended task and runtime, not only by weight reconstruction error.

If a method includes adaptive weight-training steps, name those steps rather than letting “PTQ” hide them. Classification labels vary; the actual optimization procedure is what matters.

QAT adapts the model to the representation

Quantization-aware training places deployment-like quantization effects in the forward path while retaining trainable higher-precision parameters. Optimization can move weights toward solutions that tolerate the chosen grid and clipping.

Jacob et al.'s 2017 work connects simulated quantization with integer-only inference. The training graph must quantize where inference does, including after relevant graph transformations such as batch-normalization folding. It is an early concrete example, not the origin of every modern QAT method. Jacob et al., §3 ↗

QAT can improve a poor PTQ result, but costs data and optimization. It does not guarantee recovery of full-precision performance, or make a mismatched export correct.

Fake quantization keeps a floating-point model

Fake quantization returns reconstructed floating-point values on a discrete grid. They simulate quantized operands; they are not necessarily packed low-bit storage or actual integer arithmetic during training.

For symmetric illustration, $z=0$ and $q\in[-7,7]$ give fifteen levels. This sacrifices the extra negative code of a full signed four-bit [−8,7] range. Do not label this illustration as the exact Q4_0 packing scheme.

def fake_quant(x, s, z=0, qmin=-7, qmax=7):
    # Python round uses nearest, ties-to-even.
    q = min(qmax, max(qmin, round(x / s) + z))
    return s * (q - z)

# x=0.6, s=0.25 → 0.5; x=2 → 1.75 (clipped)

The snippet shows only the scalar forward rule. A tensor implementation needs broadcasting, dtype, range validation and the backward policy. With ties-to-even, rounding before adding an odd zero point matters at exact midpoints. PyTorch affine fake quantization ↗

The backward rule is a surrogate

Rounding has derivative zero almost everywhere and is discontinuous at bin boundaries. Ordinary differentiation through that map would not provide a useful weight-training gradient.

A straight-through estimator (STE) substitutes a backward rule, commonly an identity-like gradient inside a chosen unclipped interval. A clamped surrogate can suppress gradients outside that range. These are optimization choices, not the true derivative of rounding.

Keep the forward reconstruction and backward approximation distinct. If scales or clipping bounds are learned, specify their gradients separately. Detaching a scale and learning it are different methods; a universal identity STE does not describe both.

Introduce the quantization regime deliberately. An abrupt narrow range can destroy the forward signal before adaptation begins. Monitor loss and the fraction of clipped values as observers/ranges stabilize.

Calibrate the tensors used at inference

Weights are mostly static at deployment; activations depend on inputs. A static weight range can be computed from the checkpoint. An activation range needs representative data or a runtime dynamic policy.

Choose whether observers use extrema, moving averages, percentiles or learned bounds. Freeze or update them under a versioned policy. Rare important outliers can be absent from a convenient calibration subset.

Match language, modality, context, masks, cache use and generation regime. Per-channel weight quantization and per-token activation quantization solve different grouping problems. The observer axis must match the exported kernel axis.

For convolutional models with batch normalization, folding changes weight ranges. Quantizing the unfused weights and serving fused ones breaks the numerical simulation. More generally, calibrate the graph the runtime actually executes.

Export the same quantization contract

Export includes packed values, scales, zero points, group size, signed range, layout and exceptions. Match weight-only versus activation quantization, dynamic versus fixed scales, accumulators, rounding and residual/output casts.

Run a small layer-by-layer comparison through fake-quant training, converted reference and the actual serving kernel. Then test end-to-end tasks. Differences can arise from graph fusion or layout even when the number of bits matches.

Measure memory and latency on representative shapes. Scale storage, alignment, dequantization and unsupported operation fallbacks all count. Quantizing weights alone does not necessarily relieve a cache-bound long-context workload.

MAI: a selective FP8 training recipe

The report separately retains FP32 in pre-softmax attention/router calculations, final logits, residual/MoE combination, sensitive weights, optimizer state and relevant gradient reductions. This explains why “FP8 training” is selective rather than an eight-bit state for every tensor.

Its implementation also fuses casts and describes stochastic gradient downcasts. Preserve those boundaries when reproducing the recipe. A different model or hardware backend needs its own ablation; the table is evidence about this run. MAI §2.6.3 ↗

Gemma: QAT for several deployment targets

Gemma 4 reports mobile per-channel mixed two-/four-bit weights with eight-bit activations, and a separate open block-Q4_0 target. These are not one interchangeable format. It also introduces per-block activation bounds for FP16 compatibility; bounding activations is distinct from integer fake quantization.

The original note treats these footprint figures as GB. The primary table labels them “Gb”; retain that unit-label ambiguity rather than silently rewriting the publication. Active sparse weight footprint is not the total weight memory that must reside somewhere.

The report's 150M vision encoder uses W8A8 and reports roughly 400→200 MB including on-device compilation overhead. Its 44% latency comparison also changes model/hardware context, so it is not a controlled quantization-only speedup.

The 305M audio encoder uses mixed two-/four-/eight-bit weights and INT8 activations; disk footprint is reported as 390→87 MB against an older 680M encoder. Architecture and precision both change. Preserve that denominator when reading the compression claim. Gemma 4 §2.5 and Table 3 ↗

DeepSeek: cache precision is not compute precision

DeepSeek-V4.1 describes main-cache E2M1 values with one E4M3 scale per 16 channels. It does not use NVFP4's second global scale. Cache quantization occurs after RoPE and is included in post-training QAT; local sliding-window attention retains FP8 cache precision.

The cache is dequantized before attention computation. Four-bit storage therefore does not mean that attention executes as a native four-bit product. Its reported 890 bytes per token combine architectural cache sharing/compression with precision. At one million tokens this is 890 MB decimal, about 0.83 GiB for that cache component—not total serving memory.

Engram tables and KV projections use FP8 under separate storage/placement choices. Deterministic lookup enables host prefetch/RDMA during serving, while RL places tables on GPU under different pressure. Moving a table is not the same intervention as quantizing it.

The fused Mega-mHC path includes residual input mixing, coefficients, prenorm and FP8 conversion. The reported activation-traffic term changes from $(4n+4)d$ to $(2n+2)d$; this halves that component, not all model traffic. DeepSeek-V4.1 precision and serving sections ↗

Communication is a separate boundary

A compute cast, a compressed collective and a serving export need not use the same representation. Low-bit communication changes reduction error, packing and scale ownership even if the next GEMM is higher precision.

If ranks share a scale, reduce the required maxima over the intended group. If each shard has independent block scales, preserve that metadata instead. An arbitrary global amax reduction can change a local recipe and add unnecessary synchronization.

Specify whether reduction happens before or after reconstruction, and in which accumulator type. Validate against the intended global objective. Tensor, expert and data parallelism have different ownership and error boundaries.

Distributed Training and Parallelism → owns the communication schedules. A training recipe also needs a serving comparison: inference casts, cache precision and fusion can differ.

Promote a precision change with evidence

First establish a stable reference and record the target hardware/kernel. Next select the expensive boundary: matrix arithmetic, activation memory, weights, cache or communication. Do not optimize all of them at once.

Choose format, grouping, scale policy and sensitive exceptions. Test exact casts, intermediate errors, saturation and gradients on small representative inputs. Then run a matched short training or PTQ baseline and measure task quality, memory and elapsed time.

Use QAT when adaptation is justified by that baseline. Validate the converted artifact, long contexts, rare inputs and real serving shapes. Rehearse resume with scale/RNG state. Promote only after the numerical and operational evidence agree.

The result is a versioned precision recipe, not just “use FP8.” Keep measured speedups separate from assumed peak rates, and keep the source conditions attached to every copied report recipe.

References and connected recipes

Transformer Engine 2.13 primer ↗ · Delayed scaling ↗ · MXFP8 ↗ · NVFP4 ↗ · Jacob et al. ↗ · Su Jianlin attention rounding article ↗

MAI-Thinking-1 ↗ · Gemma 4 ↗ · DeepSeek-V4.1 ↗. The floating-point bins, small block matrix, spike sequence and probability-rounding experiments are explicit explanatory models, not reported training measurements.