Training Data

The training dataset is a sampling policy

Separate the available corpus from the stream the model receives. For source s, define its unique token supply U_s, sampling rule p_s, processed exposure D_s and epoch budget. Then state whether the rule selects documents, sequences, tokens or completed agent trajectories.

A source's usefulness depends on its quality, diversity, overlap, model scale, stage and evaluation. Two individually useful sources can substitute for each other. Complementary sources can become more valuable together. Single-source scores do not determine the best joint mixture.

The task is constrained mixture design under a horizon, not the search for one universally best dataset. Scaling Laws → connects exposure, novelty and model capacity.

Keep collection and transformation lineage

Record source version, acquisition time, rights and usage restrictions, language/domain labels, extraction version and parent IDs. Preserve a route from a packed training span back to its document or generated trajectory.

Keep raw availability distinct from retained data and processed token mass. A release reporting trillions of training tokens does not establish that all tokens are distinct, permitted, uncontaminated or high quality. A generated view, paraphrase or second pass needs its own exposure count, not a new independent-document claim.

Version filters and drop priorities with the corpus. A change in classifier threshold or cross-source priority changes the experiment even if the final percentages look identical. Sensitive or disallowed data require exclusion at collection and later review; deduplication is not a substitute.

Preserve the useful structure during extraction

A general text parser can discard equations, code blocks, tables, visual context or repository relationships. MAI uses structured parsers, source-specific extractors, model-assisted processing and raw markup when conversion loses too much useful signal.

Separate low-cost screening from expensive enrichment. For an interleaved document, inspect text statistics and quality before retrieving every image. Recheck quality after images are available. Preserve the representation that carries the task: an image-text document is not always interchangeable with its text-only copy.

Remove page furniture carefully. Repeated navigation is often redundant; repeated instructions or code structure can be meaningful. Define the unit—document, paragraph, span, repository or image—before applying a rule. Template families include calculators, directory pages and form-generated stubs; different parameter values can carry distinct supervision. Inspect false removals, not just aggregate retained-token counts.

Quality depends on what must be learned

Formatting cleanliness, factual content, reasoning density, provenance, executability and diversity are separate axes. A perfectly formatted stub may teach little. A noisy-looking technical document may contain rare useful knowledge.

Learned quality classifiers scale filtering but inherit their labels and teacher biases. Test by language, source family and difficulty. A classifier trained mainly on fluent English can penalize low-resource languages or unfamiliar notation. Source-specific classifiers and calibrated thresholds reduce—but do not eliminate—that risk.

Retrieval against trusted seeds is another selection tool. Lexical and vector similarity can find related material, while quality and provenance constrain acceptance. Similarity is not truth, nor proof of novelty. Preserve difficult counterexamples when the classifier's disagreement reflects ambiguity rather than a clear labeling error.

Define redundancy before removing it

Exact identity is defined relative to preprocessing. Byte equality differs from lowercased text equality. Hashes need collision handling appropriate to the pipeline. Similarity thresholds are candidate rules, not a guarantee of interchangeable meaning.

Repeated coding exercises or explanations can be semantic review candidates. A shared subject does not establish that the reasoning, edge cases or supervision are redundant. Do not remove every repeated structure. Synthetic contrasts and controlled parameter variations may deliberately reuse a document. Their supervision can be different even when the content is the same.

Approximate overlap to find review candidates

With an ideal uniformly random permutation, the probability that two sets have the same minimum-hash item equals their Jaccard similarity. Many independent signatures estimate overlap. Locality-sensitive hashing (LSH) places likely similar signatures into shared buckets so every document need not be compared with every other document.

Take A={1,2,3,4} and B={1,2,3,5}. Their intersection contains three items; their union contains five. Under a uniformly random ordering, each union item is equally likely to come first. The minima match exactly when that first item belongs to both sets.

Three of the five possible first items are shared. Thus the exact collision probability is 3/5, not a guarantee that any single signature matches. Reuse the same ordering for both documents; independent orderings per document would destroy the comparison.

def minhash_item(items, rank):
    if not items:
        raise ValueError("A nonempty set is required")
    return min(items, key=rank.__getitem__)

LSH groups signature rows into bands. All r rows must match within a band; at least one of b bands must match to propose a pair. With independent rows, a band matches with probability $s^r$. All bands miss with probability $(1-s^r)^b$. The candidate probability is therefore $1-(1-s^r)^b$.

At similarity s=0.8, five bands of ten rows propose the pair about 43.3% of the time. More bands increase recall but also candidate volume. More rows per band make matching harder. These comparisons hold the other parameter fixed; changing either also changes signature length.

This is a probabilistic candidate gate, not a deterministic similarity threshold. Implementation hashes, dependence and shingle choices affect its behavior.

Verify candidates with the intended rule. Token sets discard order and counts; shingles retain some local order but not semantics. “cat sat on mat” and “cat never sat on mat” overlap heavily but disagree. Semantic review must also distinguish contradiction and complementary detail. Stanford CS336: MinHash and LSH ↗

Deduplication changes source ownership

Within-source deduplication removes repeated views in one family. Cross-source deduplication asks which source keeps shared material. A drop order can prefer trusted provenance, richer representation or higher-quality formatting.

If a web copy is dropped in favor of a curated copy, nominal web share no longer describes the same content inventory. A source ablation can accidentally transfer overlap into another source rather than remove it. Audit shared clusters and retained owners during comparisons.

Connected-component deduplication needs care: A≈B and B≈C do not imply A≈C. Unchecked transitive clustering can merge progressively different documents. Compare representatives, inspect cluster diameter or use a conservative review policy. Log decisions so an over-aggressive merge can be reversed.

Follow one corpus through the gates

Less redundant exposure is not new information

Deduplication usually reduces raw corpus size while preserving a chosen set of distinct material. It does not generate novel facts. Under a fixed training budget, it can make room for more distinct documents and reduce accidental overexposure.

Duplication creates a gap between raw and unique tokens. Removing duplicates reduces that gap for the retained corpus under the chosen definition. After deliberate resampling, processed tokens can again greatly exceed unique supply.

Deduplication can therefore change scaling efficiency, but the effect is empirical. It depends on what was removed, what replaced it and the training/evaluation recipe. Larger capacity can increase memorization risk; it does not make every duplicate harmful or establish one universal capacity threshold.

Measure memorization without confusing it with certainty

Repeated exposure can increase exact-string recall and narrow gains that fail to transfer. Track source-level loss distributions across checkpoints, repeated-span exposure and held-out performance. Compare unique-document splits when repeated variants are correlated.

The original MAI-related heuristic inspects improvement on tokens with extremely low negative log-likelihood (NLL). Concentrated gains can signal saturated structure or memorization. Low NLL alone is not proof: common grammar and predictable code can be legitimately easy.

Use complementary checks such as held-out unique examples and controlled extraction/repetition probes where appropriate. A stricter epoch cap is a decision to test, not a causal conclusion from one loss histogram. Preserve rare useful content instead of treating every near-certain token as waste.

Separate corpus duplicates from evaluation leakage

Training-set redundancy is an efficiency/learning issue. Evaluation contamination occurs when benchmark items, answers, solutions or close derivatives enter training or selection in a way that invalidates the intended evaluation.

Check exact and approximate overlap, parent documents, repositories and generated descendants. Split by content family or task lineage before making paraphrases and contrastive variants. Randomly splitting derived views can place the same underlying task on both sides.

Absence of a simple string match does not prove absence of leakage. Conversely, a public topic overlapping a benchmark is not automatically an exact leaked answer. State the contamination test, exclusions and remaining uncertainty. Keep a held-out final suite distinct from the metric used repeatedly to choose mixtures.

Examples, tokens and supervised targets are different weights

A long reasoning trajectory can supply most tokens while representing few examples. In SFT, user/tool/context tokens may consume compute but carry no direct imitation target. Masks change target contribution without necessarily changing input length.

Report all three quantities. A target-token share is not an input-compute share or a gradient-norm share. Packing, truncation, padding, source masks and batch construction further alter the realized stream.

Let length and masking change contribution

Loss reduction completes the sampling definition

A sequence mean gives each nonempty example the same total scalar-loss weight. A global token mean gives longer target sequences more weight. Neither is inherently the correct objective. Source-level multipliers add another axis.

A masked action can remain as context for later valid actions. Nanbeige4.2 keeps a bad turn and its observation but sets that assistant turn's imitation mask to zero. Removing the turn would invent a different history. The mask removes a direct target term; it does not automatically detach all gradients through the contextual computation.

def masked_token_mean(losses, mask):
    count = sum(mask)
    if count == 0:
        return None  # skip by the declared batch policy
    return sum(l*m for l, m in zip(losses, mask)) / count

Treat unique supply as a constraint

A small high-quality source can be useful to upweight, but it cannot fund unlimited new exposure. Epoch caps make feasible mixtures explicit. Source overlaps must be resolved before calculating unique supply.

If Σ_s cap_sU_s<D_total, no mixture satisfies the horizon. Reduce the horizon, acquire more useful material, or deliberately revise the reuse limits with evidence. Renormalizing percentages alone cannot solve the deficit.

This count is not a law of useful information. Repeated tokens can contribute less than novel tokens; a deliberate resampling policy and a model of effective data value are separate choices.

Caps are not permanent physical constants. Loss saturation, transfer, regularization, model capacity and source diversity can change the useful reuse region. Discounted repetition models → estimate a different quantity from an operational cap.

Check whether the proposed horizon is feasible

Evaluation weights are not sampling weights

A weighted held-out NLL suite supplies a stable search signal. It is often cheaper and less sensitive to output formatting than repeated generative evaluations. It still encodes chosen priorities and depends on the held-out distribution.

Some domains have different baseline NLL scales or tokenization properties. State whether metrics are raw, normalized or changes from a baseline. Otherwise a weight can have a different effective importance than its number suggests.

A scalar weighted sum is convenient, not mandatory for every multi-objective search. Constraints and Pareto methods can preserve separate metrics. A weighted sum can miss nonconvex frontier regions. Keep the domain vector alongside the aggregate.

A frontier expresses competing priorities

A mixture dominates another if it is no worse on all stated metrics and strictly better on at least one. Nondominated candidates form a Pareto frontier. A coding improvement and a general-language regression may place two candidates on the same frontier rather than reveal one overall winner.

Choose priorities and minimum acceptable domain performance before inspecting noisy candidate wins. Prefer robust neighborhoods over one fragile optimum. Uncertainty can change apparent dominance, so repeat or expand comparisons that matter.

Move the priority, not the measured evidence

Use local and global search for different decisions

Local search changes subcorpora inside a fixed family budget: code repositories, PDF buckets or mathematics subsets. Global search changes the family shares while keeping each family's internal composition fixed.

This hierarchy makes hundreds of sources manageable. It can miss cross-family interactions, so occasionally test joint changes. Maintain source caps and consistent dedup ownership in every candidate rather than enforcing them only after selecting a winner.

Train several proxy mixtures under the same architecture, tokenizer, horizon, optimizer, schedule and evaluation. Keep zero/failed/unstable results with explanations. Promote a few candidates instead of repeatedly tuning to the same proxy validation set.

Fit a surrogate without trusting its optimum blindly

The Laguna-related original proposes sampling around a manually designed prior, training a swarm of proxies and fitting capability regressors. Optimize those regressors under simplex and practical constraints, then run the proposed candidate.

A surrogate optimizer can exploit its own extrapolation errors. Penalizing deviation from the prior limits unsupported movement; it does not validate the surrogate. $D_{\rm KL}(x\|x_0)$ is infinite if x puts positive mass where x₀ has zero mass. Smooth the prior only through an explicit modeling decision, not silently.

Use held-out mixture predictions, uncertainty and measured follow-up runs. Code/math optimization can regress commonsense metrics. Preserve that trade-off instead of describing aggregate gains as universal quality.

Validate mixture rankings at larger scale

Small runs are useful for narrowing the search. They do not guarantee rank invariance. A low-diversity source can help early and saturate; a broader source can improve longer. Repetition and capacity can change the comparison.

MAI reports a STEM-heavier mixture winning small-scale comparisons but losing to a code-heavier candidate at larger scale. Validate at a larger model or longer horizon before freezing the mix. Match the intended production training density as well as model size.

A scale-up win still has a recipe scope. Changing architecture, data snapshot, tokenizer or post-training objective can require revalidation. Scaling ladders and efficiency gain → compare curves rather than one point.

Change the mixture at the right training stage

A static mix can spend scarce high-quality material too early. A staged recipe can establish broad coverage, then emphasize code/math, reasoning or long interactions. Staging is a hypothesis about learning order and reuse, not proof that rare data always belongs late.

The Smol playbook emphasizes multilingual budget trade-offs, excessive early code exposure and late high-quality sources. Nanbeige jointly schedules data quality and learning rate. MiMo treats pretraining as support for later RL exploration. Each recipe has its own goal and controls.

Count cumulative exposure across phases. Context extension, continued pretraining and SFT can repeat related material. A new stage name does not reset the data's history. Track masks and target tokens separately from context tokens.

Test a late injection from a common checkpoint

Branch a baseline checkpoint; keep part of its mixture; upsample one candidate source; run a short continuation; compare with an equally cooled baseline branch. Hold LR policy, tokens and checkpoint/optimizer state fixed where the mechanism permits.

This answers “does this source help at this stage?” It does not establish its value from initialization. If the candidate runs a different cooldown or token budget, data and optimization effects are confounded.

The Llama 3 report describes a half-trained 8B proxy, 40B continuation tokens, a linearly cooled learning rate and a 30/70 candidate/default mixture for these tests. Its separate final-run annealing section says 40M tokens; these are different reported procedures, not interchangeable budgets. Llama 3 annealing experiments ↗

MAI: novelty, priorities and rank reversal

MAI describes public/licensed human-generated pretraining data, no synthetic pretraining, AI-content detection and no packaged open-source corpora. That is this report's policy, not a universal rejection of synthetic data.

The final reported mixture is Code 54.6%, STEM 15.8%, Math 5.4%, Web 14.9%, PDFs 4.7%, Books/journals 3.1%, other multilingual 1.6%. The rounded entries total 100.1%; do not treat them as exact simplex coordinates.

Math receives 5.28× sampling and code 2.22×, while web/PDF is below one pass. The 30T stream therefore reflects selective reuse rather than 30T independent novel tokens. Extraction, multiple dedup stages, global drop order, epoch caps and scale-up mixture validation form one design loop.

The combination—not any one percentage—is the reusable method. Complete MAI profile →

Nanbeige4: quality tagging and scheduled reuse

Nanbeige4 starts with over 60 quality dimensions, reduces them to 20 after overlap/usefulness experiments, and distills strong-model labels into a scalable annotator. It reports content dimensions outperforming formatting signals and 0–9 scores outperforming binary selection in its experiments.

Lexical/vector retrieval against curated seeds complements the labels. Provenance and score constrain acceptance. The filtered pool is 12.5T, with a 6.5T higher-quality subset reused for at least two epochs; the processed stream reaches 23T. Synthetic data accounts for 15% of token mass.

This supports the combined recipe, not an isolated causal effect for every quality dimension. Nanbeige4.2 increases math/code/synthetic QA/STEM and adds some agent trajectories in a 28T corpus. Its concurrent architecture change prevents attributing all gains to data. WSD mechanism →

DeepSeek: filter cheaply before expensive enrichment

The V4.1-Flash original reports 45T multimodal training tokens and a 7:1 text-only/multimodal token ratio. This is not an image-count ratio or proof of distinct tokens. Fine-grained expert quality criteria target low-information generated text, weak-model outputs and poor translations; post-training still uses synthetic data.

Its interleaved pipeline screens document statistics, deduplication and quality before image retrieval, then repeats quality checks with visual information and SmolVLM scoring. Some rejected interleaved documents become image-text pairs. Native pairs, interleaved PDFs/webpages, OCR/grounding/pointing, long-tail data, image-code pairs and computer-use trajectories provide distinct views.

After separate processing, overlapping text documents are replaced by richer multimodal counterparts with the larger configured epoch count retained. Coordinate context-extension exposure to limit overlap. Deterministic ultralong splitting and best-fit packing report padding ≤10⁻⁴; document boundaries and attention/loss policy still matter.

The vision path uses a contrastive bootstrap on about 47B image-text pairs, then 236B generative-alignment tokens with an auxiliary 4B MoE. These are stage-specific counts, not verified unique pairs. The auxiliary model is discarded. More initial high-resolution contrastive compute yielded little final benefit in those experiments. Keep stage and modality accounting separate. Full data/vision pipeline →

GLM and Qwen: classifiers and structured source coverage

GLM-5 emphasizes learned web/world-knowledge quality classifiers, language-specific filtering, math/science quality, natural long documents and synthetic dependency tasks. Its original reports fuzzy dedup increasing unique code tokens by 28%; this needs a denominator and replacement/retention context, not the interpretation that deleting data generates tokens.

Software-engineering data connects files to commits, issues, pull requests and source state: about 10M issue–PR pairs and 160B unique retained tokens. A repository change teaches relationships that isolated files omit. Main pretraining is about 27T; 32K/128K/~200K mid-training adds 1T/500B/50B, totaling a rounded 28.55T.

Qwen2.5 reports 18T training tokens, Qwen2-Instruct quality filters, specialized math/code data and rebalancing overrepresented web domains. The primary report assigns the 40% current-maximum-length /60% shorter-sequence mix to Qwen2.5-Turbo's progressive context stages, not a universal 4K→32K mixture for every variant. Qwen2.5 data and context sections ↗

The Llama 3.1 report's approximate final mix is 50% general knowledge, 25% math/reasoning, 17% code and 8% multilingual. The older atlas's 95/5 language summary does not describe that report's final mix. Preserve source/version distinctions. Llama data mix ↗

Shieldstral: hold content fixed, vary the policy

Shieldstral maps heterogeneous labels into instruction + query + document → yes/no. Per-source processors define label/category mappings, positive/negative rules, formatting and query pools. Template paraphrases vary the wording without changing the underlying decision.

For one violent document, a violence query can be positive while an unrelated hate-speech query is negative. The document is not globally safe in the negative view. This prevents the shortcut “harmful-looking content always means yes” and trains criterion-conditioned classification.

Positives include generic risk, the annotated category, targets and category ancestors. Negatives include unrelated categories/demographics, safe content and nearby sibling categories. Positive duplication with independently sampled wording balances many negative pairings, but does not create proportionally new content.

Strict/moderate/lenient wording expresses a desired operating point; it does not guarantee a measured false-positive rate. An LLM agreement filter checks overall and category labels, but can remove ambiguous cases and transfer verifier biases. Full decision-model conditions →

Construct contrasts without multiplying independent evidence

Shieldstral's approximately 54.1M constructed samples combine 45.2M transformed public text, 4.4M synthetic contrastive text and 4.5M multimodal samples. Approximate shares are 83.5%, 8.1% and 8.3%; rounding and correlated views matter.

The synthetic taxonomy has 11 superclasses and 73 leaves derived from 11 taxonomies. A generator rewrites safe text toward a target category while avoiding a sibling. It preserves source language, uses English queries and reports temperature 0.7. The same rewrite receives a positive target query, negative sibling query and ancestor positives.

The multimodal pipeline combines violation data, clean negatives, general image data, query mutations and hard negatives. About 2,000 phrasings cover 14 visual subcategories; approximately 30% use inverse wording. A visual reranker filters mismatches with looser thresholds for scarce violations and stricter thresholds for abundant negatives.

The 23.3-point F1 gain accompanies a large recall increase and reduced precision. It is not an unqualified safety improvement. Samples share content, templates, hierarchy and generators. Split by lineage, check deployment calibration and distinguish in-domain policy transfer from unrestricted novel-policy understanding.

Agent data includes the environment and verifier

A plausible task prompt with an unreliable verifier is poor RL data. Nanbeige4.2 reconstructs a parent repository commit, dependencies and a container; hides the reference patch/tests; requires fail-to-pass target tests and pass-to-pass regression tests. Successful patches and individual turns receive separate quality filters.

Current-model failures guide later repository/task selection. Several scaffolds vary prompts, editing and context policies; the model should learn repair behavior rather than one driver's conventions. Full trajectories retain history; invalid calls, loops, redundant actions and truncation receive explicit filters or masks.

Its hybrid tool data combines persisted Python tools, live MCP services and model-simulated tools. A task generator creates instructions and Python verification. Repeated solver attempts guide difficulty changes in chain depth, parameter inference, retrieval and tool composition.

Office data starts from clustered multi-file artifacts. A sandboxed synthesis agent produces tasks, rubrics, trajectories and deliverables; an independent judge validates outputs before recycling them. The loop expands coverage but risks self-reinforcing generated artifacts.

Task breadth, execution, graders and filtering belong to the data object. Their complete environment and RL recipes remain in the Nanbeige4.2 profile →, not a pretraining token-mix formula.

Task latency must not define the training mixture

MiMo's original task mix is coding 68%, general tool use 12%, visual/aesthetic 13%, context following 3% and cybersecurity 4%. The report's unit is a task mixture; do not reinterpret those numbers as supervised token or gradient shares without additional accounting.

Across 25 profiled sources, generated tokens vary by 90× and active rollout duration by 66×. Equal concurrency tends to return more fast tasks. The Sample Mixer uses adaptive source concurrency, deficit-aware target scheduling, predicted KV/inference demand and limited replay after startup/recovery.

This separates desired batch composition from latency-induced arrival composition. Monitor accepted completions, rejected/truncated attempts, tokens and replay counts. Equal target task counts still do not mean equal compute, target tokens, advantages or update contribution.

Flash's pretraining split is 26T text/22T omni-modal; Pro's is 27T/3T. Context grows 32K→256K→1M. The same task under different harnesses is a distinct interface view, not a new independent task. Weighting harness families is a coherent experiment, but it is not a reported result merely because source mixing exists.

Freeze a versioned recipe, not just percentages

Record corpus/parser/filter versions, duplicate cluster ownership, unique supply, example sampling, input and target token shares, masks/reductions, caps, stage budgets, checkpoint lineage and held-out splits. Log realized exposure as well as requested shares.

Test local/global candidate changes, source overlap sensitivity, repeated-data saturation and scale-up ranking. Preserve domain vectors and uncertainty alongside aggregates. Confirm downstream adaptation and agent/environment behavior when those are the target.

Keep failed runs and rejected data decisions auditable. Change one mechanism at a time for causal tests; retune jointly for a recipe comparison. Collection, mixture and optimization cannot be certified by a high retained-token count or one winning proxy.

References and connected recipes

Llama 3 ↗ · Qwen2.5 ↗ · Shieldstral ↗ · Nanbeige4 ↗ · Smol Training Playbook ↗ · Laguna report ↗