Pretraining: Experiments and Run Design

Define what the run must achieve

Write three decisions before choosing a model: why train, what capabilities matter, and how the model will be used. A code base model for local deployment has different constraints from a general multilingual assistant.

Specify task families, languages, context, latency, memory, licensing, hardware and deadline. Define evaluation sets and hard constraints. A weighted held-out negative log-likelihood (NLL) objective is useful for cheap development, but it does not replace generation, execution or safety checks.

MAI explicitly combines Coding, STEM, Math, General and Multilingual evaluations with weights 0.5, 0.175, 0.175, 0.1 and 0.05. These are capability priorities, not training-data proportions. Do not copy them without the same objective. MAI §§2.2–2.3 ↗

Predeclare what would justify another experiment and what would justify freezing. Exploration consumes the same calendar and compute needed to finish the run. Prefer high-leverage questions about data, scale or a bottleneck over many tiny improvements with uncertain transfer.

First compare training from scratch with continuing an existing checkpoint or using an existing model. Training is justified only if the capability, rights or deployment requirements warrant its cost. State acceptable failure risk, privacy/audit requirements, checkpoint cadence and replacement-node policy alongside hardware and deadline.

Build a baseline you can trust

First make a cheap known-good configuration reproducible. Fix tokenizer, data snapshot, packing, masks, objective reduction, initialization, optimizer, schedule and evaluation. Verify the distributed implementation against a small reference.

A baseline is a versioned object. Record its code, configuration, random seeds, loader state and measured cost. An accepted change becomes the next baseline; do not compare new experiments against several silently different recipes.

Use direct intermediate checks. Decode fixed examples and inspect repetitive output, special tokens, reasoning modes and tool formatting. Such checks reveal bugs that NLL can miss. They are debugging evidence, not a substitute for held-out quantitative evaluation.

Inspect system prompts and multi-turn continuations explicitly. A missing system message, broken chat template or style failure can leave aggregate loss nearly unchanged. Interactive inspection helps find these bugs; a few good examples do not establish general capability.

Choose what the proxy preserves

Two cheap experiments answer different questions. Train the intended architecture for fewer tokens to check early behavior and implementation. Train a smaller model to explore many alternatives. Neither automatically predicts the final ranking.

Preserve the mechanism being tested. A routing experiment needs representative experts and allocation. A data experiment needs representative mixture, repetition and token horizon. A context experiment needs the attention operator, positions and masks used at deployment.

A small positive result is a candidate for promotion, not proof. A negative result can be a useful filter, but delayed learning or a scale interaction can reverse it too. Reject a broken implementation confidently; reject a research idea only within the tested regime.

High-quality specialized data can help a larger model while harming a small proxy. Conversely, a source can look useful early and become repetitive later. The original Smol material uses arXiv as an example of capacity-dependent value; treat the observation as recipe-specific.

Match the resource that answers the question

Equal parameters, equal tokens and equal FLOPs are not always simultaneously possible. State the comparison rather than call it fair without a denominator. Count active versus stored mixture-of-experts (MoE) parameters, embeddings, attention and recomputation consistently.

Use repeated seeds or paired branches where practical. Preserve failed runs and report uncertainty. A tiny loss difference may not exceed run noise. Validate interactions after isolated ablations: a better tokenizer, mixture and schedule can change one another's ranking.

Architecture labels can hide budget changes. Untying embeddings adds a separate output matrix. Head sharing changes key/value projection capacity. Changing the feed-forward expansion changes both stored capacity and arithmetic. Recalculate the budget before attributing a quality gain to the mechanism.

A ladder tests scale transfer

Train several sizes under a defined family and compare curves. Keep the tokens-per-active-parameter (TPP) regime fixed for a particular ladder when that is the intended experiment. Use a second horizon if the final run is much more heavily trained.

Keep architecture and optimizer families, the data objective and the training-density rule consistent within the ladder. Changing any of them changes the comparison. Width, depth, optimization, data saturation and systems overhead can alter rankings as scale grows.

MAI uses 100–200 TPP for many architecture ablations and discusses 500–1,000 TPP runs. Its label “near Chinchilla” should not become the universal 20-TPP reference. Its reported 30T/35B final backbone exposure is about 857 TPP; the active-parameter denominator matters.

Hold out a larger confirmation point. A fitted curve can conceal poor extrapolation. Data-mixture rankings can invert with scale: MAI reports a small-scale STEM-heavy winner losing to a code-heavier mixture at larger scale.

Scaling Laws → owns frontier fitting and allocation. Here the ladder is a decision gate: an idea progresses only when the intended regime supports it.

Convert a loss improvement into equivalent cost

Efficiency gain asks how much the baseline would spend to match the candidate's loss. $\operatorname{EG}=1.3$ means 30% more baseline cost than candidate cost. The candidate saves $1-\frac1{1.3}\approx23.1\%$ against that equivalent baseline, not 30%.

Take an illustrative candidate cost of 100 units and an equivalent baseline cost of 130 units. Both costs refer to the same target loss.

For a normalized example, use $f(C)=1+C^{-0.5}$. At $C'=4$ and $L'=1.4$, equivalent baseline cost is 6.25 and $\operatorname{EG}=1.5625$. These assigned numbers illustrate the inverse, not a measured model.

Fit FLOP-based and time-based curves separately. MAI's shared-expert MoE-every-layer comparison reports weighted $\operatorname{EG}_{\rm FLOPs}=1.03$ but $\operatorname{EG}_{\rm Time}=0.82$ against its interleaved baseline. The latter says the actual stack loses the modeled advantage. It does not prove the architecture can never be optimized.

Inverse estimates become fragile near the floor and outside the fit range. Include fit uncertainty, cost convention and the scale range. Do not infer an equivalent cost for $L'\le E$ from this equation.

Promote an experiment through explicit gates

Choose experiment order by expected information per resource. Screen dimensions with a plausible large effect first: mixture, architecture shape, optimizer/schedule and a dominant system bottleneck. A tiny theoretical gain can cost more in implementation and delayed start than it returns.

The gates are not a mandatory six-run ceremony. Reuse evidence when its conditions match. Write the unresolved assumption at promotion rather than hide it behind a passing benchmark.

Useful early candidates include embedding tying, attention head sharing, depth/width and feed-forward shape, optimizer/scheduler families, learning rate, batch size and mixture weights. Choose a candidate because it tests an unresolved mechanism, not because the sweep is easy to automate. Change one component or a small declared group while fixing the remaining baseline.

Set an exploration deadline and a promotion threshold before seeing the results. Stop when further uncertainty reduction is worth less than the delay and reserved run resources. This is a resource decision, not proof that the chosen recipe is globally optimal.

Freeze a recipe, not just hyperparameters

Record model shape, tokenizer, special tokens, data manifests, mixture/caps, packing/masks, optimizer groups, precision and schedules. Include batch semantics, parallel layout, checkpoint format, evaluation harness and the serving conversion.

Freezing limits uncontrolled variation; it does not forbid responding to a correctness incident. Record any authorized intervention as a new run branch with its parent and reasons. Keep a recovery plan and a conservative fallback.

Prefer a broad stable optimum over a narrow fragile win when gains are uncertain. Warmup–stable–decay (WSD) offers a long extendable stable phase and a later decay branch. Cosine can be appropriate for a fixed horizon. Neither is universally superior. Optimization →

Small high-quality datasets can be reused deliberately with measured caps. Reuse is not infinite novelty. Reserve resources for annealing, context extension, recovery, final evaluation and the required post-training stages.

Start from committed model compute

The 6PT rule approximates dense non-embedding forward/backward arithmetic. Attention, embeddings, sparse structure and other operations can matter. Full activation recomputation can raise executed arithmetic toward an 8PT approximation in some settings.

Do not charge recomputation twice. Model FLOP utilization (MFU) uses useful model arithmetic rather than all recomputed hardware work. If the measured rate is already a model-FLOP rate, its time penalty already appears there. Hardware-FLOP rate needs the executed-work model instead.

A peak specification multiplied by an assumed MFU is a scenario, not a measurement. Use a representative full-stack rehearsal and a lower/base/upper range. Distinguish a healthy-step rate from an end-to-end rate that already includes failures and checkpoints.

A 256-GPU run-design worksheet

The worked constraint is 256 GPUs for 90 days. One feasible allocation is 8% exploration, 7% rehearsal, 75% main training and 10% contingency. These shares sum to 100%; independently chosen percentage ranges may not.

At the default assigned rate, the 7B scenario gives about 6.40T tokens under the approximation. That arithmetic is not a recommendation to train for that many tokens. Check corpus novelty, repetition, inference economics, final context and other stages first.

If context extension shares the main window, split the compute explicitly between lengths. If post-training is in the same allocation, reserve it before spending the main budget. Do not claim a base-training token budget that uses someone else's reserved resources.

For the coding scenario, verify the accelerator variant and interconnect, whether existing weights can be continued, and whether the deadline includes experiments. Evaluate licensed repository context, documentation and permitted issue/patch histories, not just isolated files. The last chronological checkpoint is not automatically the best export.

Choose capacity and exposure together

Several parameter/token pairs can fit the same approximate compute. A larger model learns from fewer tokens under a fixed budget. A smaller, more heavily trained model may reduce recurring inference cost, even off the pretraining-compute frontier.

Compare dense and MoE only after accounting for total state, active arithmetic, communication, serving behavior and implementation maturity. A large stored parameter count does not determine the token budget or guarantee a quality advantage.

Use scaling estimates to narrow candidates, then verify actual throughput and data limits. A 3B on-device target is not chosen by the same objective as a large hosted coding model. Lifecycle allocation →

Make the architecture executable

Specify depth, width, feed-forward expansion, activation, normalization, initialization, vocabulary, tying and attention. Grouped-query attention (GQA) can reduce KV-cache cost; its training speed and quality still require a matched test. Choose positions and masks for the intended lengths.

For coding, audit tokenization of indentation, whitespace, operators, Unicode, separators and fill-in-the-middle controls. Measure tokens per source/language. A larger vocabulary changes embedding memory and projection cost.

Architecture and kernels are coupled. A new operation can change fusion, communication, activation storage and the best layout. Prototype correctness before measuring speed. Transformer Architecture → · Attention → · Tokenization →

For a conservative dense recipe, specify AdamW, decoupled weight decay, warmup and decay, clipping, packing, selected high-precision operations and residual initialization. Test alternatives through a ladder. Optimizer settings, architecture and kernel implementation form one experiment, not independent defaults.

Version the stream and its safeguards

Keep source rights and provenance through parsing, quality filtering, safety checks, deduplication, decontamination and tokenization. Freeze the resulting manifests and held-out splits. Track raw inventory, useful unique tokens and expected passes separately.

For code, handle repositories, forks, generated/vendor files, licenses, secrets and build/test validity. Repository and temporal separation can reduce leakage. A passing test is not proof that generated code is safe or licensed.

Define token-based mixture weights, source caps and realized exposure. Document probability, sequence probability and target-token share differ. Loss masks and reduction affect gradient contribution too. Audit domain losses rather than rely on one global average.

Training Data → owns curation, exposure, mixture search and source-conditioned evidence. The run design records exactly which version of that policy the loader executes.

Register immutable raw snapshots with timestamps, source, license and lineage. Audit correctness, coherence, information density, authority, parse integrity, difficulty and harmful content separately. For synthetic data, record the generator and verifier. Deduplication needs a declared unit and threshold: documents, spans, repository forks and evaluation overlap are different decisions.

Do not remove legitimate repeated conventions or rare-domain coverage merely to maximize a deduplication rate. Version fixed, temperature-based, quality-weighted or adaptive sampling policies. Record each mixture transition with consumed tokens and checkpoint lineage. Evaluate both specialization gains and general-domain regression.

Treat input delivery as part of the run

A correct stream can still leave GPUs idle. Use sequential shards, local caching, prefetch, decompression and bounded queues appropriate to the storage system. Measure CPU saturation, queue depth, cache misses and input wait time.

Checksum and validate shards. Quarantine a corrupt input with an explicit policy and logged source impact. Unrecorded skipping changes the experiment. Backpressure must not silently reorder or duplicate samples.

Compare a synthetic resident input with the real loader to isolate a bottleneck. Faster synthetic steps implicate the input path, but not one particular storage component. Keep the actual loader cursor, sampler, mixture counters and packing buffers in the restart contract.

Make batch and memory accounting explicit

Tensor, pipeline, context and expert ranks are not automatically additional independent examples. Count the data-parallel replicas that own distinct batch portions. Specify whether loss is token-mean or sequence-mean before accumulating gradients.

Memory includes parameters, gradients, moments, any master copy, activations, communication workspaces, temporary tensors and allocator headroom. Precision and sharding change each term. A universal bytes-per-parameter estimate does not cover this full state.

Measure peak memory at representative length and batch, not only steady allocations. Use activation recomputation/offload and optimizer/state sharding under an explicit trade-off. More accumulation can fit memory while lengthening the step or changing overlap.

Increasing batch size changes gradient noise, update count and communication frequency. Retune learning rate and token schedule when needed, rather than treating higher utilization as free optimization improvement. Long-context activations can dominate where short-context optimizer state previously set the limit.

Map ownership onto the cluster

Place frequent large communication on the fastest links where possible. Map tensor, pipeline, data/state, context and expert parallelism to actual device groups and node topology. Do not apply a generic “3D parallelism” label without explaining ownership.

For each collective ask: which ranks participate, what bytes move, how often, which dependencies wait, what overlaps, and what happens to the slowest rank? Pipeline bubbles, expert imbalance and cross-node reductions need different fixes.

Benchmark the real shape and kernel stack. More devices can reduce efficiency through smaller GEMMs or added communication. Distributed Training and Parallelism → owns dimensions, sharded matrices and scheduling mechanisms.

Choose a framework with a real rehearsal

Megatron, DeepSpeed or an internal stack can each be appropriate under different needs. Framework reputation does not settle the choice. Pin versions and test the intended workload rather than repeat an old speed comparison as a current ranking.

Read the execution path before adopting the stack. Trace an optimizer update, a distributed checkpoint and a restart. Repeat the trace with MoE, context parallelism and custom numerics enabled. Clear abstractions help locate throughput drops and distributed bugs; a simple baseline alone does not establish that clarity.

An internal framework gives direct access to its maintainers, but also assigns maintenance and recovery to the organization. An external framework brings community support and established implementations, but its abstractions can constrain changes. Identify an owner for critical bugs and upgrades before the run.

Mature stacks often trade ease of modification for operational coverage. Smaller stacks can shorten experimentation, but require more validation before a long run. These are tendencies, not current product rankings. Compare implementation time, debugging effort, instability and runtime cost on the same representative workload.

MAI's YOLO sharding annotations describe ownership; they do not automatically inject all communication. That design makes synchronization explicit. It is not a promise that every annotation-based framework has the same semantics.

Features, correctness, stability and recovery interact, but their product is a metaphor, not a measured dimensional throughput equation. A stack without an indispensable feature is infeasible even if its simple baseline is fast.

Increase operational risk in stages

1. One-device correctness: verify forward loss, masks, targets and gradients.
2. Small sharded run: compare reductions, optimizer updates and precision.
3. Multi-node run: test topology, collectives, data delivery and stragglers.
4. Soak run: inspect thermal behavior, memory growth and long-lived processes.
5. Actual-scale rehearsal: measure throughput, checkpoint stalls and restoration.
6. Main run: start from the frozen manifest with owners and response gates.

Exercise resume before committing the expensive run. A checkpoint that has never been restored is an untested artifact. Fault injection must use an authorized test environment, not interrupt a production job without a plan.

Reserve real resources for rehearsal. A long run can expose bugs that a few healthy steps cannot. The Smol playbook's tensor-parallel seeding incident illustrates why initialization and rank state deserve early checks; the retained original places the restart around one trillion tokens.

Pay for long context where it matters

At fixed sequence count, dense attention grows quadratically with length. At fixed total tokens, there are fewer sequences, so its per-token arithmetic grows roughly linearly with length. These are different comparisons.

Train the broad representation at a cheaper length when the objective permits. Then extend with validated positions, masks, packing, learning rate and data. Test long-range use, not only whether the model accepts a larger input.

A staged plan uses short or moderate main training, then an intermediate extension and a final target-length stage. Mid-training names a position in the pipeline; it is not necessarily supervised fine-tuning. Repacking keeps source proportions while exposing longer documents, such as repositories, books and technical articles. It changes sequence construction, not the underlying source identity.

Repacking the same broad mixture can reduce a distribution change, but it does not preserve every attention relation. Document boundaries, loss masking and cross-document access still matter. Long-context NLL, retrieval, generation and short-context regression are separate tests.

Choose the main length from measured throughput and memory on the intended stack. Reserve extension compute separately. Changes in length also affect communication, activation storage and checkpoint cost. A favorable arithmetic estimate does not guarantee the final latency or quality.

Compare staged and always-long compute

With fixed token shares qᵢ and lengths Lᵢ, sum T·qᵢ·(a+bLᵢ). Do not apply a quadratic length multiplier to every token when the comparison holds total tokens fixed. The static default is 1.115 cost units staged versus 2.6 always-long.

Read the context recipes with their scope

The Smol material describes 4K→32K→64K training, with roughly 50B-token context stages and fresh schedules. The Smol ablation finds specialized books/articles/synthetic long data no better than its broad baseline in that experiment. The suggested reason is that the baseline already contains long web/code documents.

In that Smol experiment, fresh extension schedules were better than attaching the stages to the tail of the main decay. This comparison motivates a separate adaptation trial, not a universal schedule rule. Positional calibration, distant retrieval and attention behavior can all change; NLL alone does not identify which mechanism improved.

For its tested extrapolation, 64K→128K with YaRN works better than 32K→128K; 256K degrades. Adjusted base frequency (ABF) in rotary positional embeddings (RoPE) and YaRN scaling need joint tuning. These findings do not certify every model at those lengths. Positional Encodings →

MAI Appendix B reports most extension NLL improvement in the first 1–10% of iterations. Its proposed calibration explanation is a hypothesis, not proof that no new capability is learned. The Appendix's final recipe states 140B tokens at 256K; the main phase table uses 150B at 262,144. Keep both source scopes visible rather than silently reconcile them.

Preserve retrieval and generative QA tests alongside NLL, short-context quality and per-domain checks. Fast adaptation in one report is evidence for a staged trial, not a universal extension budget.

MAI reports a staged-versus-always-long comparison and limited short-context degradation under its tested recipe. That supports measuring a cheaper staged candidate, not assuming equivalence in a new model. Keep a broad mixture and objective where appropriate, but explicitly test short-context regression after extension.

The connected Smol development loop

The playbook starts from a compact deployment goal: a 3B dense Llama-style model for multilingual, coding and mathematical use. It connects architecture and data selection to the later assistant pipeline rather than optimize only pretraining loss.

GQA provides a practical cache trade-off. Hybrid no-positional-embedding (NoPE)/positional attention and intra-document masking are separate choices tested for the intended context behavior. AdamW and flexible schedules provide an operational baseline. Learning-rate and batch scaling rules narrow a search; they do not replace testing.

Data priorities include preserving English while adding languages, code and math. Early broad exposure and later high-quality material have different roles. Define these changes by tokens and actual loss masks, not just example counts.

The complete sequence is objective → cheap baseline → ablations → recipe → full run → late data/context stages → SFT → preference optimization → selective reinforcement learning (RL) → serving evaluation. A change earlier in the chain can alter a later stage's behavior. Smol Training Playbook ↗

Smol treats multilingual allocation as a budget trade-off with English and other domains. In its recipe, excessive code early can hurt general evaluations. Broad English-web exposure comes first; higher-quality code/math and reasoning/Q&A enter later. These are recipe-specific observations, not evidence that one source order is universally optimal.

Its short-context results do not settle long-context architecture: intra-document masks can matter more at longer lengths, and hybrid RoPE/NoPE has to be tested at the intended lengths. GQA balances cache cost against quality within a particular model comparison, not a guaranteed quality ranking for every model.

Branch late data changes from one checkpoint

A long stable phase makes late annealing experiments inexpensive relative to retraining from scratch. Save a common parent, inject a candidate source and compare matched decay branches. Keep parent optimizer state, token counts and evaluation fixed where the question requires them.

This answers whether the source helps at that stage. It does not estimate its value throughout pretraining. A candidate can win through repetition, formatting or altered mixture rather than broad new knowledge. Check held-out domains and exposure.

Price the candidate's collection, filtering and implementation cost too. A narrow gain that delays the full run may be a poor project decision. Keep both successful and unsuccessful branches for the next recipe.

Keep the base-to-assistant contract intact

Establish a supervised fine-tuning (SFT) baseline before adding more optimization. Fix chat templates, turn boundaries, special tokens, reasoning controls, tool schema and which tokens receive loss. Formatting is part of the checkpoint contract.

The Smol pipeline uses offline preference approaches such as DPO/APO before deciding where RL is needed. Reasoning and direct-answer modes have different length distributions and evaluation needs. Do not treat one mode's benchmark gain as every mode's improvement.

Verifiable-reward RL can improve task success while increasing output length. Reward design, termination and length pressure matter. A larger scalar reward or longer answer is not automatically a better deployed policy.

Keep matched base-checkpoint adaptation checks when feasible. Later pretraining is not always better for every transfer task. Checkpoint selection through transfer →

Complete SFT reference → · Preference methods → · Hybrid reasoning modes →. The connected notes explain the objectives and implementation contracts.

The initial SFT “baby baseline” checks mode switching, preserved system prompts and multi-turn reasoning before optimizing benchmark scores. In the Smol account, DPO is an offline baseline and APO improves reasoning in its comparison. This does not establish that either objective dominates on all preference data.

The /think and /no_think modes change response length, not only the prompt label. Smol reports that verifiable-reward RL on concise mode can still produce long reasoning unless the reward constrains length. Measure correctness and length policy separately for each mode. Long traces can dominate token volume despite being few examples; count supervised targets as well as examples.

Two frontier schedules and their boundaries

MAI's midtraining also changes STEM/math/code emphasis. A context change and a mixture change must not be interpreted as one isolated context ablation. Its recipe's AdamW, dropout, clipping and precision details remain in the complete report and the connected optimization/numerics owners.

DeepSeek-V4.1 reports 100.6M global batch tokens, a 2,000-step warmup, LR $2.6\times10^{-4}$ through 28T, cosine to $2.6\times10^{-5}$ by 40T, then constant through 45T. At a constant reported batch, $\frac{45\times10^{12}}{100.6\times10^6}\approx447{,}316$ updates; that is not GPU-hours.

It also separates backbone pretraining from frozen-backbone DSpark drafter training. Later DSpark updates track the changing backbone without propagating its objective into the backbone. Speculative Decoding → owns the drafter mechanism.

Shared attention state across pipeline boundaries, shadow indexers, microbatch lifetimes, row-sharded Engram tables and prefetch affect run feasibility. Separate vision work and context-rank image balancing also belong to the operational recipe. DeepSeek §§2.4.3, 3.1, 4.2 ↗

MFU, hardware utilization and goodput differ

Hardware FLOP utilization can include recomputed operations. MFU does not automatically price startup, rollback or time outside the measured stepping window. A run can show healthy kernel utilization and still waste substantial allocated time.

Duration goodput measures time relative to an explicit ideal reference. Checkpoint pauses, replay, restart, degraded placement and silent slowdowns reduce it. It is not automatically GPU utilization or an absolute measure independent of the chosen reference rate.

Track committed progress separately from attempted updates. A dashboard that counts repeated work as new tokens can falsely show a fast run. A sustained MFU regression is worth investigation even without a crash.

Design the durable checkpoint transaction

Store weights, optimizer moments, schedules, scaler, steps/tokens, random-number generator (RNG) streams, sampler/loader cursors, mixture and packing state. Include topology/sharding metadata, code/version lineage and FP8 scaling history when used.

Asynchronous saving overlaps work with I/O, but it must snapshot a consistent logical step. Retain buffers until writes finish. Distributed shards need a complete commit manifest, checksums and a verified readable checkpoint before declaring durability.

A later incomplete save must not replace the last valid recovery point. Define retention, replicated storage and read validation. Checkpoint portability across different layouts is a separate capability from restoration on the original layout.

For replicated checkpoint state, one validated storage read followed by a broadcast can reduce many workers reading the same bytes. Sharded state still needs its correct owners. Bound the broadcast traffic and verify version, completeness and checksums before using the restored state. Less storage pressure does not waive the checkpoint transaction.

Hot standby can reduce startup latency by replicating appropriate state and managing replacement workers. It does not remove communication, correctness or state-ownership obligations. Watchdogs and coordinated failure handling must avoid divergent ranks continuing incompatible steps.

Watch a restart preserve or change the run

Save at step 3; fail after step 5; restore step 3; execute steps 4–6; commit step 6. The logical run advances six steps while hardware executes eight updates. Full state matches the uninterrupted toy reference. Resetting only moments or only data/RNG changes the result.

The independent reference's final state is cursor=6, RNG=11, m=715 and weight accumulator=1350. Exact state matching here establishes only the toy recurrence. Real distributed validation compares sample IDs, target counts, loss, gradients, updates and backend-specific tolerances.

Specify the reproducibility contract

Bitwise replay requires stable arithmetic order as well as restored random state. Reduction trees, kernel algorithms, routing tie-breaking and collective topology can change floating-point results. Pin the conditions the stack actually promises.

Statistical equivalence is weaker and often more practical across different hardware/layouts. Define acceptable numerical and behavioral tolerances, evaluation seeds and what differences trigger investigation. Do not claim exact replay after changing topology unless it is verified.

Compare uninterrupted and restored branches before the large run. Check FP8 history, tensor-parallel initialization/seeds, sampler position and complete optimizer state. Hashes are useful for exact contracts; matching one loss scalar is not enough.

Balance checkpoint pauses against replay

Short intervals cost more saving time; long intervals lose more recent work. For assigned c=30 seconds and M=6 hours, I*≈1,138 seconds, or 19 minutes. The two overhead terms are each about 2.64% there.

This is a planning approximation, not a guarantee. Startup downtime, asynchronous interference, correlated faults, varying save cost and failure during saves require a richer model. With fully hidden I/O, effective blocking c can be small, while bandwidth and snapshot memory still constrain frequency.

Measure replay, checkpoint stalls and restore latency separately. Do not count the same replay in both an end-to-end throughput rate and a second goodput discount.

Continue with fewer devices only under a contract

Gemma 4 describes slice-granularity elasticity on TPUs. After a localized failure, its larger models continue on fewer slices. The report describes reconfiguration delays falling from many minutes to seconds. This is source-specific infrastructure evidence, not a generic GPU recovery promise. Gemma 4 §2.7 ↗

During a repair window R, reconfiguration time d and surviving throughput fraction r give approximate useful healthy-equivalent time r·max(0,R−d). Waiting idle gives zero work during that window. For assigned R=600s, d=20s and r=0.75, the degraded run earns 435 equivalent seconds.

That comparison assumes the same valid logical objective, no rollback and no extra rejoin cost. Charge repartitioning, data ownership, batch changes, communication inefficiency and restoration too. Changing replica count without preserving the intended global batch and reduction can change the experiment.

Join model and system telemetry

Retain logs preceding an event. A loss spike, throughput drop and source-ratio change at the same step can guide investigation without identifying the cause. Compare a controlled branch before making a permanent recipe change.

Use tail latency, not only averages. One slow rank can hold the job. Assign an owner and response path to each important alert; a dashboard without action criteria is not a recovery system.

Use evaluation gates throughout the run

Run cheap fixed held-out NLL frequently. Schedule richer task suites and expert inspection less often. Use a complete final audit after any required post-training. Version prompts, dependencies, sampling, execution environments and scorer behavior.

For coding, include completion, infilling, repository editing, compile/execute checks and private held-out tasks. Use temporal/source separation and investigate suspicious benchmark gains. Add security, license and long-context checks where the product needs them.

Predeclare continuation and stop criteria: persistent regression, instability, unacceptable leakage, degraded goodput or an unreachable deadline. Do not select a metric after seeing which checkpoint wins. Statistics and Evaluation →

Maintain an evaluation registry of prompts, solutions, repositories and known derivatives. Check exact and approximate overlap before tokenization; temporal holdouts depend on trustworthy timestamps. Include next-token loss by language/source, code memorization, long-range dependencies and expert review. Version the harness so a benchmark change is not mistaken for model progress.

Respond to an incident with evidence

Preserve the preceding durable checkpoint, failing batch lineage and logs. Decide whether the run can safely continue while collecting evidence. Check inputs, resume state, masks/reductions, precision, gradient/update scales and hardware changes.

Reproduce a short branch with one intervention and a matched control. A lower learning rate can suppress a symptom while leaving a broken loader or collective untouched. Unrecorded skipped updates alter the token/schedule contract.

For low throughput, separate input wait, arithmetic, communication, bubbles and checkpoints. For unstable loss, separate source composition from domain degradation. For a failed resume, verify complete state before blaming optimization.

Training Stability and Generalization → contains clipping order, loss regimes and checkpoint diagnostics. The run plan supplies who responds, which evidence to retain and how recovery fits the remaining budget.

Replay the failing batch and neighboring batches from the same verified checkpoint, then compare a known-good batch. Data-correlated failure points toward the input path; persistence across batches warrants numerical and optimizer-state checks. Neither pattern identifies the cause by itself. Preserve the environment and use fewer ranks when a faithful reproduction is possible.

Quarantine a repeatedly failing node. Compare elastic continuation with waiting, including reconfiguration and rejoin costs. Compiler graph breaks, excessive logging and critical-path synchronization can reduce throughput without a hardware fault. Sunk tokens do not justify continuing a compromised trajectory; restart or stop when verified mitigation cannot protect the objective.

SmolLM3: a running job can still implement the wrong experiment

SmolLM3 exposed four different faults: cold-shard eviction, a loader index with excessive shared-memory use, insufficient sequence shuffling, and correlated tensor-parallel initialization. Smooth loss did not reveal the last fault; matched tensor-parallel and nonparallel runs did.

Each fault breaks a different contract. Diagnose the broken contract before changing optimization.

Do not translate “shared memory” into an unverified GPU page-fault explanation. Do not fix every random-number stream by assigning arbitrary rank-specific seeds. Replicated tensors must agree; partitioned tensors need the intended logical initialization. Dropout and sampling have their own stream contracts.

A useful regression test compares a small reference model with its distributed form at initialization, forward loss, gradients and one update. Specify tolerances and intended stochastic differences. Then test a full-horizon loader configuration and a full-corpus soak run separately.

Smol Training Playbook: The Training Marathon ↗ · Distributed ownership → · Version the stream →

Write the comparison before launching the sweep

Cheap experiments are useful only if they preserve the mechanism that could change the ranking. SmolLM3 used target-size 3B mixture ablations and later branched annealing tests from a shared checkpoint. Smaller proxies do not automatically preserve data rankings.

Use one-factor changes to locate a cause, then test interactions before freezing the recipe. Keep a ledger of failed runs and tuning budgets. A method that received ten learning-rate trials should not be compared with one untuned baseline without saying so.

For late data changes, branch candidates and a baseline continuation from the same parent. Match continuation exposure and schedule. For RL, also version environments, judges and accepted-sample rules. Otherwise a better score can come from an easier world rather than a better policy.

Smol Training Playbook: data-mixture and annealing experiments ↗ · Packing-aware budget example → · Task versus interface transfer →

Validate the exported artifact

Convert the selected checkpoint under a versioned procedure. Validate tokenizer, control tokens, positions, attention masks, cache layout, precision and weight ownership against the training reference.

Run the actual serving stack, not only the training evaluator. Compare logits or outputs under an appropriate numerical contract, then test latency, memory and required context lengths. A faster kernel or quantized export can change behavior.

Keep the base, post-trained and serving artifacts distinct, with their lineage and evaluations. A completed training job is not a validated deployment. Finishing the run means producing a recoverable and interpretable artifact whose measured behavior meets the declared objective.

References and connected notes

Smol Training Playbook ↗ · MAI-Thinking-1 report ↗ · DeepSeek-V4.1 report ↗ · Gemma 4 report ↗

Chinchilla ↗ · Kaplan scaling ↗ · Megatron-LM ↗ · Megatron distributed training ↗ · ZeRO ↗ · Reducing activation recomputation ↗ · The Pile ↗ · Deduplicating training data ↗ · DataComp-LM ↗ · JAX scaling book ↗

The worksheet, context bars, checkpoint interval and integer recovery trace are assigned explanatory models, not measured training results.

MAI recipe → · DeepSeek recipe → · Training Data → · GPU Infrastructure → · Performance Accounting →