Compute, Memory, and Performance

Which resource limits this workload?

A slow workload does not tell you which resource would make it faster. A larger accelerator can leave a small decode request almost unchanged. More ranks can reduce local computation while increasing communication. High GPU activity can coexist with little committed training progress.

Start with three quantities: the work to perform, the bytes to move, and the storage needed while that work executes. Then match those quantities to the hardware and the measurement window. Each estimate should expose an assumption you can test.

This note connects operator arithmetic to a complete resource plan. Ideal bounds help reject impossible estimates and choose an investigation. They do not replace measurements of the full stack. Profiling explains how to test the resulting bottleneck hypothesis.

Define the result and its measurement window

Round-trip time (RTT) measures a request and return path. It is one latency definition, not a synonym for every operation’s latency. Report p90, p95 or p99 with the workload and sampling method. An average can hide a small group of very slow requests.

A change is useful only relative to the target. Increasing tokens per second while violating the latency service-level agreement is not a successful serving optimization. Likewise, faster attempted training steps need not imply faster durable progress.

Latency, throughput and in-flight work

Little’s law connects averages under a stable flow with consistent boundaries. W includes service and waiting, not only queue delay. Count the same items in L and λ; do not mix requests, tokens and batches.

At 100 completed requests/s and 0.2 s average residence time, average in-flight work is 20 requests. This is an accounting relationship, not proof that adding twenty workers will sustain that rate.

As offered load approaches capacity, queues can grow and tail latency can worsen. The shape depends on arrival, service and scheduling distributions. There is no universal exponential latency curve. Beyond sustainable capacity, an unbounded queue has no stable average to which this calculation can be applied.

A request that waits for several parallel services can finish only after its required slowest dependency. More fan-out can expose tail behavior even when each service’s average is small. Measure the complete dependency path.

Batching trades shared work against waiting and larger service intervals. Distinguish the delay to assemble a batch from the time to execute it. A batch of 100 messages can amortize fixed overhead while delaying the first message.

An optimization changes a specific part of the path

A cache removes some repeated work when it hits. A content delivery network can shorten a network path. Protocol choices can reduce particular setup or serialization costs. None guarantees a universal speed ranking independent of the workload.

A queue decouples producers from consumers; it does not create consumer capacity. Batching amortizes fixed work and can reuse weights. Horizontal scaling adds independent capacity only when ownership, state and shared dependencies allow it.

Memory access is often much faster than a disk seek, but “disk has low throughput” is too broad. Sequential storage bandwidth, random-access latency and a cache hit describe different operations. Compare the relevant pattern and boundary.

For example, Redis can serve an in-memory cache and a CDN can serve edge copies. HTTP/3 and gRPC change different protocol and serialization boundaries. Treat these as candidates to measure, not interchangeable guarantees of lower latency.

Orders of magnitude are not current specifications

The following rounded values are historical mnemonic estimates carried by the original reference. Their hardware, date and measurement methods are unspecified. Use them to remember different scales, not to size a current system or assert a benchmark.

A mechanical seek does not describe an SSD. A bandwidth-derived serialization time excludes propagation, queueing and protocol overhead. Compression depends on content and CPU. Replace a mnemonic with an explicit specification or representative measurement before using it in a capacity plan.

Speed and scaling answer different questions

Performance describes speed and resource use for a specified workload. Scalability describes how those results change as work and resources increase. They are related, but neither can be inferred from one lightly loaded measurement.

A query that takes 5 s even for one user suggests a slow-path investigation. A query that takes 100 ms for one user but 10 s at 1,000 concurrent users suggests a load-dependent investigation. These are assigned examples, not measured service results. Either case can involve inefficient code and shared resource limits.

Do not compare different token counts as if they were fixed-work speedups. Throughput scaling at an equal latency/quality constraint is another useful comparison. Report the baseline, resource count, workload and cost.

More resources can expose another limit

The next limit may be a serial stage, shared database, network link, rank imbalance or small per-device matrix. Heterogeneous workers can also create stragglers. More total peak compute does not remove these limits automatically.

Redundancy can consume capacity and add replication work. A system need not have zero redundancy overhead to be scalable. Stateful systems can scale through explicit ownership and partitioning; statelessness simplifies some paths but is not a universal requirement.

Fixing unnecessary quadratic work can be more effective than adding servers. However, optimization and scaling are not a mandatory sequence: deadlines, availability and engineering cost matter. Choose the intervention whose predicted effect addresses the observed constraint.

Check p95/p99 under representative load and inspect shared dependencies. In model training, relate speedup to local shapes, communication and pipeline bubbles. Distributed Training and Parallelism owns the layout and schedule mechanisms.

A slow query can motivate algorithm changes, indexes, caching or faster local hardware. A load-dependent bottleneck can motivate load balancing or database sharding. Optimized hardware and commodity clusters are deployment choices, not definitions of performance and scalability. Distributed Services and System Design explains the ownership, cache and replication mechanisms.

Start with the tensor contraction

For a two-input contraction, multiply the sizes of distinct dimensions. Count a shared batching axis once and a contracting axis once. A shared output axis indexes independent subproblems; a shared axis absent from the output is summed.

For batched matrices X[B,M,K] and W[B,K,N], the count is 2BMKN, not 2B²MKN. An elementwise product or outer product without reduction does not acquire a factor of two merely because two operands occur.

def matmul_work(m, k, n, qx=2, qw=2, qy=2):
    flops = 2 * m * k * n
    traffic = qx*m*k + qw*k*n + qy*m*n
    return flops, traffic, flops / traffic

For positive integer m, k and n and positive byte sizes, this educational function counts ideal one-time input reads and one output write. It excludes bias, beta·Y, conversion, scales, tile reloads and intermediate work. The byte model and operation model must describe the same call.

Why forward plus backward costs about 6PT

For $Y=XW$, backward computes $dX=dY\,W^\top$ and $dW=X^\top dY$. The gradients match their operand shapes. Each matrix product has the same leading operation count. Together they cost about three times forward: $6MKN$.

The shortcut neglects several operations and quadratic attention. Frozen operands, selective gradients and selective checkpointing change the count. Input embedding lookup is not a dense vocabulary matrix product. Output unembedding is a dense projection and can matter, especially for large vocabularies.

Tied input/output weights reduce stored parameters, not the output projection’s arithmetic. State whether P includes that projection. For mixed attention, recurrent layers or sparse models, count the actual operations before applying a global shortcut.

Rematerialization trades saved intermediates for repeated work. If measured useful-model FLOP/s already includes its slowdown, do not apply a second 8/6 time penalty to that rate. Count executed work when using an executed-hardware rate.

Count the projections the architecture actually uses

Query and output projections each contain D² entries. Key and value each contain D·Hkv·K entries. Multi-head attention gives 4D². Grouped-query attention reduces K/V projections and cache size; it leaves Q/O unchanged.

A two-projection FFN has an up and down matrix. A gated FFN also has a gate matrix. Using an old 2DF rule for a three-matrix SwiGLU block undercounts the projection work.

Count layers, vocabulary projection, shared experts and unusual projections separately. Shape-based accounting should also state biases and small operations that were neglected. “Exact per layer” is exact only within that declared operation inventory.

Long context adds work that parameter counts miss

For MHA, a gated FFN with $F=4D$ gives $P_{\rm layer}\approx16D^2$. The full-pair attention-to-linear ratio is then $\frac S{8D}$. Parameter-only estimates increasingly undercount attention as context grows. Other FFN ratios and head layouts change the crossover.

Causal attention has $\frac{S(S+1)}2$ useful pairs, not exactly $\frac{S^2}2$ at small S. A causal-aware kernel can skip masked work; a dense masked implementation may still execute it. FlashAttention reduces intermediate traffic but does not make all required attention arithmetic disappear.

The bar shows assigned arithmetic shares, not time shares. A bandwidth-bound operator can dominate time while occupying little of the FLOP bar. Use the complete forward/backward kernel behavior to turn this inventory into a timing hypothesis.

MoE separates stored capacity from active work

An MoE expert matrix is stored for each expert, but a token executes only its selected experts. For E experts and top-k routing, stored FFN parameters scale with E; leading token-level expert work scales with k.

Record total parameters, active parameters, E, k and any always-active shared expert. Use total state for checkpoint and persistent-memory capacity. Use active executed matrices for the first-order arithmetic estimate.

Routing, padding, token dropping, imbalance and expert communication remain separate costs. A group can store experts that receive little work while another expert sets the step’s critical path. Arithmetic alone cannot establish efficient sparse execution. Mixture of Experts connects those routing and systems mechanisms.

Turn arithmetic and traffic into resource bounds

Below the ridge, the bandwidth ceiling is lower. Above it, the compute ceiling is lower. These describe the two-resource ideal model, not proof that the measured kernel saturates either resource.

For ideal resource times a and b, max(a,b) describes full overlap and a+b describes serialization. Actual execution can exceed their sum because of launch, dependencies, extra traffic or ineffective instructions. Do not call the sum an upper bound on the real program.

Compare measured throughput with the bound under the same counting convention. A point far below the roof motivates another investigation: insufficient reuse, small tiles, instruction mismatch, synchronizations or a different memory boundary. The roof does not identify that cause by itself.

Operational intensity is another name for arithmetic intensity when the counted byte boundary is the same.

To build a useful roofline, first choose the operation, precision and memory boundary. Then count its FLOPs and boundary-crossing bytes. Obtain matching hardware ceilings, compute intensity, and compare measured performance with the bound.

cuBLAS, fused kernels and FlashAttention are candidates for their supported operations. They are not interchangeable replacements for arbitrary code. Choose a change that addresses the measured operation and preserve its numerical contract.

Match precision, instruction path and memory boundary

Match the arithmetic to the precision and instruction path. Scalar FP32, TF32, BF16 Tensor Core and FP8 peaks are different ceilings. Structured-sparse specifications cannot be applied to an arbitrary dense matrix.

An Iridge calculation needs matching compute and bandwidth specifications. The historical example below uses the 40 GB A100, not an unspecified A100 variant. Do not mix its FP32 roof with BF16 weight arithmetic.

Increasing the relevant compute roof while holding bandwidth fixed moves the ridge right. That does not make memory-bound work faster. Lower-precision weights can instead reduce bytes; whether the compute path also changes is a separate question.

HBM, L2, shared memory, registers and inter-GPU links have distinct boundaries. An HBM roof counts HBM traffic, not every register access. Cache residency and tiling can change which boundary supplies the useful constraint. Use a relevant specification and a representative measurement rather than one universal GPU roof.

NVIDIA’s June 2021 A100 data sheet ↗ lists 19.5 TFLOP/s FP32 and 1,555 GB/s HBM bandwidth for the 40 GB PCIe and SXM variants. Dense BF16 Tensor Core peak is 312 TFLOP/s; the 624 value requires sparsity. These are manufacturer ceilings, not achieved measurements.

Under the weight-dominated BF16 estimate, batch 32 has intensity about 32, below 200.64. It does not cross this BF16 roof. The FP32 ridge cannot justify the opposite conclusion. Extra activation/cache traffic and inefficient kernels can lower achieved performance further.

Reuse changes the bytes, not the mathematical result

On the assigned 1,555 GB/s FP32 bandwidth roof, the add ceiling is about 129.6 GFLOP/s. At $I=0.25$ it is about 388.8 GFLOP/s, approximately 2% of 19.5 TFLOP/s. These are bounds from assigned traffic, not measured kernel results.

Fusion can keep a producer’s intermediate on chip instead of writing and rereading it from HBM. Tiling can reuse input blocks. Better coalescing can improve achieved bandwidth. Each intervention needs actual traffic and latency evidence; a theoretical one-read model can be unattainable for a finite working set.

Occupancy is not an independent objective. More registers or shared memory can lower residency while eliminating expensive traffic. Conversely, high residency does not guarantee useful throughput. Profiling and Kernel Optimization distinguishes these measurements.

Batching moves intensity toward the compute ceiling

For a small dense decode batch, leading forward projection work is roughly $2BP$ FLOPs while BF16 weights contribute roughly $2P$ bytes. If weight traffic dominates, $I\approx B$ FLOP/byte. Batch 1, 8 and 32 therefore have approximate intensities 1, 8 and 32—not universal compute-bound thresholds.

Input, output and KV-cache traffic keep the exact intensity below a weight-only estimate. Large token rows can reuse weights, but a limited matrix width or finite tiles constrain that benefit. Weight quantization changes traffic; scales, unpacking and the compute implementation can change the time too.

Prefill exposes many prompt-token rows and often larger GEMMs. Single-user decode often has little weight reuse. Long-context decode can instead be dominated by KV access. Batch, context, architecture and implementation determine the regime; phase names alone do not prove the bottleneck.

When comparing prefill and decode pools, preserve their distinct latency, memory and utilization requirements. A simplified BF16 weight model must use a matching BF16 compute roof, not the lower FP32 example to claim a real crossover.

Batching needs shared weights

Batching helps because several rows reuse one matrix. Merely grouping independent operations does not create that reuse.

Compare $X[B,D]W[D,F]\to Y[B,F]$ with $X[B,D]W[B,D,F]\to Y[B,F]$. In the second contraction, each row has its own matrix. Both perform the same leading arithmetic, but their weight traffic differs.

For a weight-dominated shared matrix, $I_{\rm shared}\approx B$. With distinct matrices and large widths, $I_{\rm per\ row}\approx1$. The batch factor cancels in the second ratio.

The table is ideal accounting, not a benchmark. Cache residency, tiling and conversion can change actual traffic. In MoE, reuse depends on rows routed to the same expert, not the total input batch.

How to Scale Your Model, Rooflines: independent per-row matrices ↗ gives the one-byte variant. This comparison uses two-byte storage consistently.

MFU, HFU and calendar progress have different numerators

Model FLOPs utilization (MFU) estimates useful model arithmetic relative to the declared peak. Hardware FLOPs utilization (HFU), under this convention, includes recomputed arithmetic. A GPU-busy percentage or occupancy measures something else.

End-to-end progress also loses time to checkpoints, failure, replay, recovery and degraded placement. Count accepted logical tokens only once. A replayed step can use the hardware effectively while adding no new durable progress.

Pair healthy-step efficiency with a calendar productivity measure and an explicit reference rate. If an end-to-end measured rate already includes downtime, do not discount it again. If executed work exceeds the declared peak interval, inspect units, FLOP convention, precision and timing instead of clamping the ratio to 100%.

MFU=1 means the declared useful-model FLOP rate equals the declared peak. It does not prove that every operation or every calendar interval is efficient.

Use the same FLOP convention when comparing kernels, parallel layouts or regressions. High GPU-busy percentage can coexist with low useful arithmetic rate. High healthy-step MFU can coexist with poor durable progress. Follow run design and restart contracts for the operational checks. Frontier training recipes connect this distinction to the Llama 3 and MAI reports.

Reconcile a utilization claim before interpreting it

Consider eight devices, each assigned 10¹⁵ FLOP/s peak, and a 0.40 s step. Their available peak work is 3.2×10¹⁵ FLOPs. Keep this denominator fixed while checking the claimed numerator.

The original 1.2×10¹⁶ numerator gives 375%, not 37.5%. A consistent 37.5% example uses 1.2×10¹⁵ model FLOPs. The change is explicit; a ratio above one calls for an input/convention investigation, not clamping.

Under these stated conditions, the second example gives about 46.9%. Both examples are assigned arithmetic, not measured training runs.

Small matrices, launch cost, exposed communication, routing imbalance and rematerialization can reduce MFU. Larger matrices can amortize fixed costs and improve achieved kernels. A smaller model’s lower MFU does not alone prove worse engineering.

Longer context does not inherently lower correctly counted MFU. It adds useful attention work and changes shapes, memory and kernels. Compare conventions and measured step time rather than treating every changed ratio as an efficiency regression.

A compute allocation is a scenario with units

Use consistent units before choosing a model. A billion is 10⁹; a trillion is 10¹². TFLOP/s is 10¹² FLOP/s, not a quantity of work. One day is 86,400 s; 30 days are 2.592×10⁶ s; 90 days are 7.776×10⁶ s.

An illustrative allocation has 256 H100 SXM devices, an assigned dense BF16 peak of 989 TFLOP/s each, 40% MFU and 90 days. Its nominal model-work budget is about 7.875×10²³ FLOPs. This is scenario arithmetic, not a measured run.

Reserving 25% of that allocation for other experiments, integration, evaluation, failures and finishing leaves about 5.906×10²³ FLOPs for the main run. Define whether that reserve already contains downtime. Applying both a failure reserve and the same downtime factor would charge it twice.

Hardware peak and achieved utilization must match precision and density. A sparse or FP8 peak does not represent dense BF16 arithmetic. The highest uncertainty is often the achieved full-stack rate; rehearse representative work to narrow it.

For decimal scientific units, thousand, million, billion and trillion mean powers 3, 6, 9 and 12 of ten. PFLOP/s and EFLOP/s mean powers 15 and 18 FLOP/s. Keep intermediate arithmetic unrounded; round the planning result only after cross-checking it.

A feasible curve is not an equal-quality curve

Equal compute does not imply equal quality. Unique usable data, optimization risk, model capacity, serving cost, context and post-training needs constrain the choice. The rounded token counts need not match the envelope to the last digit.

Use scaling experiments to compare feasible candidates, not the resource calculator to infer accuracy. Scaling Laws connects data exposure, capacity and lifetime economics. For MoE, use an explicit active-work model alongside total-state memory.

Check the same plan in tokens and steps

For the same assigned 256-device, 40%-MFU case, useful rate is about 1.013×10¹⁷ FLOP/s. A 30B model uses about 1.8×10¹¹ FLOPs/token under 6P. The predicted healthy throughput is therefore about 563,000 tokens/s.

A four-million-token batch takes about 7.1 s. Training on 3.28T tokens takes 820,000 optimizer steps. Their healthy duration is about 67 days, consistent with roughly 75% of a 90-day allocation.

Check the plan both ways: total arithmetic divided by useful rate, and step count times measured step duration. If they disagree, inspect token definition, padding, accumulation, stage changes, rounding and timing windows. A healthy-step estimate needs a separate calendar allowance.

Count independent data, not cooperating model ranks

Do not multiply by tensor- or pipeline-parallel degree. Those ranks cooperate on the same replica’s data. A global device count is not necessarily the data-parallel degree.

With $B_{\rm micro}=1$, $S=8192$, $G=16$ and $D_{\rm dp}=32$, the batch is 4,194,304 tokens. At 563,000 tokens/s, predicted healthy step time is about 7.45 s. This is a different batch from the four-million-token example, not a contradiction.

State whether padding and masked positions count as processed work, training targets or useful tokens. Packing can reduce wasted positions. Accumulation changes optimizer-step count without multiplying the number of independent parameter replicas.

Build state memory from its actual components

This layout is conditional. Some stacks omit master weights, keep gradients in FP32, quantize states or hold extra buffers. Enumerate the implementation rather than treating 16 B/P as a universal constant.

If every state component were perfectly sharded over D ranks, its persistent share would be $\frac{16P}D$ bytes. Real layouts shard different components differently. A gather can materialize a larger weight tensor temporarily even while persistent storage remains sharded.

Use total stored MoE parameters here, not only active parameters. Include metadata, alignment and actual distributed ownership. Decimal GB means 10⁹ bytes; GiB means 2³⁰ bytes. Do not switch units silently.

A model can fit persistently and fail at peak

A coarse activation term scales with microbatch sequences, context, layers and hidden width: $B_{\rm micro}SLW$. The coefficient depends on saved tensors, backward kernels, checkpointing and sharding. Naive attention can add quadratic score/probability storage.

FlashAttention-style kernels avoid materializing the complete attention matrix. Selective or full checkpointing reduces retained values while adding recomputation. Neither policy can be inferred from parameter count alone.

Estimate a candidate analytically, then measure a representative full block and complete step. Leave explicit headroom for collective buffers, workspaces, allocator fragmentation and unexpected shapes. A configuration with only a few percent spare capacity is fragile, not a proven safe deployment.

Inference also stores the past

Training activations and inference caches have different lifetimes. Autoregressive inference retains keys and values so that later queries can attend to earlier tokens without recomputing their projections.

The cache formula uses KV heads, not query heads. At fixed layer count, head width and precision, changing 32 KV heads to 8 reduces this payload fourfold. This comparison does not claim equal model quality or measured speed.

Mixed local/global layers require a sum over each layer’s retained length. Latent caches, shared prefixes, cache sharing and quantization metadata need their own inventory. Paging, reserved capacity and temporary buffers change allocated memory. Serving Models explains cache ownership and scheduling; Attention explains the head layouts.

Count the bytes on the relevant path

Collective startup, dependency readiness, topology and contention add time. Match byte volume to the effective path bandwidth and message regime. Bidirectional totals and per-direction bandwidth must not be mixed.

Some communication overlaps backward computation. Sum of individual collective durations is not automatically exposed step time. Inspect critical-path ranks and the work that depends on completion.

Possible changes include group placement, parallel degrees, larger useful compute per collective, overlap, pipeline scheduling, sequence parallelism and expert balance. Each changes another constraint too. Follow Parallelism for the ownership mechanisms; the communication implementation requires separate evidence.

Storage bandwidth is only part of a checkpoint pause

Checkpoint storage includes weights and the state required by the restart contract. A 30B BF16 model is about 60 GB; the illustrative full 16 B/P state is about 480 GB. Distributed formats, manifests and temporary duplication add requirements.

At an assigned aggregate 20 GB/s, transferring 480 GB takes at least 24 s. The observed pause can also include consistency, synchronization, metadata and filesystem behavior. Storage serialization time is not the complete save or restore time.

If a 60 s pause occurs once per 1,800 s elapsed cycle, its fraction is 3.33%. If 1,800 s instead means useful work between pauses, the fraction is $\frac{60}{1860}\approx3.23\%$. State the interval convention.

Asynchronous saving can hide some pause but needs a consistent snapshot and completion before retirement. A file write is not automatically a durable recoverable checkpoint. Pretraining: Experiments and Run Design owns the restart transaction and validation.

Short pauses and rollback compete for useful time

For C=1 minute and M=360 minutes, the approximate interval is $\sqrt{720}\approx26.8$ minutes. Longer intervals reduce checkpoint frequency but increase average rollback. The objective is productive time, not the fewest pauses.

This simple result assumes an approximate failure model and small overheads. Restart, correlated failures, asynchronous saves, contention and interval conventions require refinements. Uniform failure position within a checkpoint interval motivates $\frac I2$; it is not guaranteed for every failure process.

At scale, use job-level failure behavior rather than only one device’s reliability. Include save, rollback, restore and reconfiguration. A failure rehearsal can test the operational contract even when healthy-step MFU looks strong.

Small average payload can still hide a slow input path

At four bytes per token ID, 3.28T stored token IDs occupy about 13.12 TB. Reading that payload once over 67 days averages roughly 2.27 MB/s. That small average does not prove a loader is fast enough at every step.

Millions of small objects, remote metadata, parsing, decompression, shard imbalance, reshuffling and shared-storage contention can create stalls. Prefetch and dynamic sequence construction also affect tail delivery. Repeated exposure or extra fields change the payload model.

Plan storage for immutable raw and normalized data, tokenized versions, manifests, optimizer checkpoints, evaluations, logs, profiles, temporary saves, retention and disaster recovery. Average payload, request rate, CPU work and burst/tail behavior are separate constraints.

Inspect the path that supplies the next batch, not only its monthly byte total. A full-stack rehearsal connects this estimate to input delivery and run design.

Separate a hypothetical rate from a total cost

256 accelerators for 90 days give 552,960 accelerator-hours. Rates of 3, 6 and 10 units per accelerator-hour give approximately 1.66M, 3.32M and 5.53M units. These are scenario costs, not market quotes.

Add CPU hosts, network, storage, support, evaluation, idle allocation, failed runs and engineering. Reserved capacity can bill unavailable or unused time. For owned infrastructure, distinguish purchase cost from amortized cost and include power, cooling, facilities and operations.

Use a sensitivity range and an explicit accounting boundary. Lower healthy step time does not automatically reduce a fixed reservation bill, although it can create capacity for additional useful work.

Serving volume changes the model-size decision

At high serving volume, extra training of a smaller model can be preferable if it meets the quality target. At small volume, the one-time training budget can dominate. Neither outcome follows from model size alone.

Include latency, memory, required parallel group, batch distribution and long-context behavior. A model that trains efficiently can still require an expensive serving layout. Inference-aware scaling provides the model/data allocation view; serving mechanisms determine the recurring rate.

Do not reuse one utilization estimate across every phase

Short-context pretraining, long-context continuation, post-training rollouts and serving do not share one resource profile. Context can add quadratic attention, reduce microbatch size and change achieved kernels. Data supply and evaluation also change.

For each stage, count its operations, estimate attainable memory/batch, measure its rate, and allocate its elapsed time. Sum the stage budgets under consistent useful versus executed conventions. Do not extrapolate a short static GEMM’s utilization to a complete long-context job.

For long-context continuation, budget suitable long documents, retrieval/extrapolation evaluation and short-context regression tests. Different evidence answers different questions: arithmetic feasibility, system throughput and model quality remain separate.

A compact resource worksheet and cross-check

Check these common failures: wrong sparse/precision peak; applying MFU twice; confusing total and active MoE capacity; multiplying batch by model-parallel ranks; missing accumulation; treating 6PT as exact at long context; charging remat twice; perfectly dividing every memory term; omitting gathers/buffers; mixing GB/GiB; calling healthy MFU calendar progress; spending the entire allocation on the main run; quoting an undated price.

A complete estimate answers how long, whether it fits, what the batch means, which communication remains exposed, how restart works, and what the cost includes. State the least trustworthy assumption and a test. Consistent arithmetic makes a design plausible; representative execution and evaluation determine whether it works.

For communication, record collective frequency as well as the largest payload. For memory, measure allocated and reserved peaks separately. For restart, include restore and reconfiguration, then validate the plan with an injected-failure rehearsal. These checks test the assumptions that simple arithmetic cannot establish. The largest uncertainty can be achieved MFU, unique usable data volume, full-scale stability or overlap. Name it and choose a rehearsal that reduces it.

Qwen: optimize the resource that each phase actually uses

The same architectural change has different costs in training, prefill and decode. Qwen makes that distinction explicit. A FLOP reduction is not enough if the implementation adds a memory bottleneck or a long tail of tiny kernels.

Attention prefill measurements use the last 16K-token chunk, batch size 1. Decode uses batch size 4 and next_n=4, including three additional MTP steps. Both selector and sparse core attention count in the attention-module comparison. These are not full-prompt prefill timings or end-to-end generation multipliers. The figure does not provide enough hardware detail to turn the ratios into a universal device forecast.

Deterministic n-gram addresses are known from tokens before Layer 2 runs. That permits prefetch while Layer 1 computes. A single lookup layer avoids repeated transfers at several depths. As a capacity illustration only, 51B parameters at two bytes each occupy 102GB decimal before optimizer states or metadata. The report does not declare that hypothetical payload to be the actual table storage precision.

Why equal-size sharding is wrong for Muon work

For K=8, a 4096×4096 matrix costs about 2.20 trillion orthogonalization FLOPs under this estimate. A 1024×16384 matrix contains the same number of parameters but costs about 0.55 trillion. These are calculated arithmetic examples, not measured step times.

Canzona separates logical optimizer ownership from physical tensor layout. A static partitioner assigns whole matrices across data-parallel ranks using estimated NS cost. An asynchronous Micro-Group pipeline reconstructs each matrix with fused all-to-all across tensor-parallel ranks. The owner then performs an update mathematically equivalent to unsharded Muon. Existing ZeRO-1 bucket geometry and reduce-scatter/backward overlap remain intact.

After semantic splitting, one layer can contribute roughly a hundred submatrices. Their updates become launch-bound. Capturing the step in a CUDA graph reduces dispatch overhead without turning it into one fused kernel. Validate full-matrix equivalence, reconstruction buffers, matrix ownership, critical-rank time and resume state before measuring throughput.

The full system still needs model weights, KV/recurrent states, residual buffers, lookup storage and optimizer state. The report does not disclose a complete cluster topology or production GPU-hours. Collectives and ownership → · Semantic optimizer splitting → · Compute claims →.

Qwen3.8-Next architecture report, §§2.1–2.3 and 3.1, Figures 1 and 6 (26 August 2026) ↗.

Sources and connected topics

• How to Scale Your Model: Rooflines ↗; Transformer accounting ↗; Training ↗; Inference ↗.
• Training Compute-Optimal Large Language Models ↗ and Scaling Laws for Neural Language Models ↗. Quality versus resource allocation.
• Megatron-LM ↗; Efficient Large-Scale Training on GPU Clusters ↗; ZeRO ↗. Parallel execution and state partitioning.
• The Optimum Checkpoint Interval for Restart Dumps ↗. First-order checkpoint interval.

The interactive examples are original ideal accounting models, not measured hardware traces or reconstructed benchmarks. Continue with Attention for operator/IO mechanisms, Precision and Quantization for numerical contracts, and Speculative Decoding for reducing serial generation work.