Small and On-Device Model Recipes

Which resource must be small?

Separate four quantities: stored parameters, parameters used in a token's pass, layer applications per token, and state retained across tokens. A sparse model can have low active compute yet require all experts in memory. A looped model stores one stack but executes it repeatedly.

Also separate one-time training from repeated inference. Overtraining can be rational when a small deployed model serves enough requests. It does not make the original training inexpensive. Compare total lifecycle cost under the target workload, not just parameter count.

Account for the vocabulary

An input embedding has V rows and d columns. An untied output head adds another V×d matrix. When the transformer body is small, these matrices can consume a large fraction of the parameter budget.

Embeddings are not inactive or useless for reasoning. They represent input tokens and support output discrimination. The budget question is whether this vocabulary and width use scarce storage effectively. Changing the tokenizer also changes token counts, sequence lengths, multilingual coverage and teacher/student alignment.

With 166,144 entries and width 3,072, two matrices contribute 1.020789B parameters. Adding a hypothetical exact 3B body gives 4.020789B. This explains the rounded Nanbeige label; it is not a tensor-by-tensor checkpoint count. Its current configuration declares untied embeddings. Nanbeige configuration ↗

Liquid: hardware and repeated exposure

The earlier interview covers models from 350M to 24B parameters, with cars and phones as deployment examples. It prioritizes memory limits, focused tasks such as tool use and summarization, and latency. A stored parameter budget and factual knowledge capacity are related but not interchangeable.

Liquid's LFM2 architecture combines gated short convolutions and attention. Short convolutions keep a bounded local history and suit some edge kernels; attention retains selective access to earlier tokens. Faster kernels on a stated device do not establish universal superiority over GQA or sliding attention.

LFM2.5-350M extends the 350M model's pretraining from 10T to 28T tokens and adds large-scale RL. Its release targets extraction, structured output and tool use. This is a version-specific budget, not the exposure of every LFM2 model. Liquid release ↗

A frequently cited compute-optimal reference uses about twenty training tokens per parameter: around 7B tokens for 350M parameters. The 28T release exposure is 4,000 times that token budget. This compares exposures, not training FLOPs or a universal optimum. A small deployment can justify heavy one-time training, but training is not automatically cheap. Scaling Laws →

The earlier interview note describes a failure-control recipe: sample five diverse responses and one deterministic response, judge them, and use a looping response as a rejected preference example. “Doom looping” means continued repetitive generation until termination or the execution budget; a finite run does not literally produce infinitely many tokens. Preference Optimization → explains how DPO learns from the preferred and rejected responses. The interview also describes verifiable rewards and n-gram repetition penalties. Those interview details have no linked primary protocol in the original; treat them as reported guidance, not a reproduced release recipe.

A looping candidate is not automatically a valid preference pair. The positive must satisfy the task, and the rejection must reflect the desired behavior. A model that repeats until its budget ends can fail an executable verifier, but neither DPO nor a repetition penalty guarantees termination.

Tools can supplement limited internal knowledge. They introduce their own retrieval, latency and reliability limits. Recursive processing can avoid placing an entire document in context, but lost evidence and failed tool calls remain possible. Using a teacher vocabulary can simplify distillation alignment while increasing vocabulary storage. The original 63% embedding and 90% effective-parameter claims lack checkpoint-specific counts; use the budget equation rather than generalizing either.

The interview emphasizes knowledge and tool-use evaluations. The release names GPQA Diamond and τ²-Bench, rather than the earlier shorthand “T2Bench.” Liquid recommends this checkpoint for extraction and tool workflows, not math, code or creative writing. External search can extend available evidence but cannot guarantee more reliable knowledge than a larger model. Validate returned evidence and recursive summaries before using them.

Nanbeige 4: invest in the training stream

The original Nanbeige4-3B is a conventional dense Llama-style model, not a looped transformer. Its main intervention is a training pipeline: 23T tokens, large reasoning SFT, answer refinement, dual-level distillation and staged RL. The later 4.2 comparison identifies roughly 4B total and 3B non-embedding parameters for both releases.

The original $\frac{23\times10^{12}}{3\times10^9}\approx7{,}667$ density uses the named 3B denominator. With rounded 4B total, the ratio is 5,750. Neither is training FLOPs or unique corpus size. Always state the denominator when comparing these figures.

The corpus includes web, scholarship, books, code and synthetic QA, textbooks, lectures and long reasoning. Synthetic material supplies 15% of final token exposure. Quality tagging starts with more than sixty dimensions and retains twenty. A strong annotator supplies labels that a smaller model learns to apply at scale.

Content-related labels outperform formatting labels in this recipe; scores 0–9 outperform binary judgments. Lexical/vector retrieval against trusted seeds complements intrinsic quality scoring. Provenance and learned quality scores constrain which retrieved samples are trusted. The pipeline retains a 12.5T pool and a 6.5T higher-quality subset, then deliberately repeats that higher-quality subset for at least two epochs to reach 23T exposure. Repetition changes training weight; it does not create new independent documents. Nanbeige 4 report ↗

Align the data curriculum with FG-WSD

Warmup–stable–decay changes the learning rate. Fine-Grained WSD also changes the data mixture inside the stable phase. Earlier broad exposure supports coverage; later higher-quality mass emphasizes the chosen capabilities. A constant LR during that transition avoids changing two variables at once.

The stages sum to 23T. During decay, Adjusting Base Frequency extends context to 64K. This is a trained extension, not proof of equally strong retrieval at every position.

The experiment uses 1T stable-phase exposure plus a fixed 100B-token decay corpus. WSD shuffles a 1:1 high/medium-quality mixture; FG-WSD uses 750B mixed tokens then 250B high-quality tokens. GSM8K, CMath and BBH use 3-shot evaluation; the other three tasks use 5-shot. The largest gains occur on the targeted math/reasoning tasks. The authors connect these gains to reasoning density in selected data. This supports the combined schedule/curriculum, not a universal best data ordering.

Base evaluation also uses a common downstream treatment: three public SFT datasets, 500K examples each, two epochs and matched hyperparameters across three bases. The bases are Nanbeige4-3B-Base, Qwen3-4B-Base and Qwen3-8B-Base. Nanbeige generally yields the strongest reasoning checkpoint. Nine runs reduce reliance on one favorable dataset. The comparison still emphasizes reasoning, where this pretraining mixture was optimized.

Find a good answer, then reconstruct its trace

Cold-start reasoning SFT uses about 30M cleaned examples at 32K: roughly 50% mathematics, 30% science and 20% code. Scaling from about 0.5M to 35M examples keeps improving the reported AIME/GPQA results. This does not prove that all small-model tasks benefit equally from that scale.

The overall 64K SFT mix broadens to 40% math/subject reasoning, 30% general QA/writing, 20% agents and 10% code. The construction pipeline separates answer search from the trace used to supervise the student.

1. Create a task-specific checklist and generate candidates with teacher models and the current model.
2. Evaluate correctness, completeness, executability, consistency and safety.
3. Critique and revise the answer, then repeat checks.
4. Condition chain completion on the instruction and refined answer; create a short summary and a coherent reasoning trace.

The final target joins reconstructed reasoning and the refined answer. An offline revision loop is not recurrent transformer depth. A plausible reconstructed trace also does not prove that it records the teacher's actual internal reasoning.

Correct student failures at two levels

Dual-Level Preference Distillation uses positive responses from Nanbeige3.5-Pro and negatives from the student. Retain pairs with a clear quality difference. Student-generated negatives focus supervision on states the deployed policy actually visits.

A DPO-style sequence margin says which answer should win. Teacher-distribution supervision supplies local alternatives on both positive and negative prefixes. Distilling on a wrong trajectory is not instructing the student to repeat its sampled wrong token: the target is the teacher's distribution at that prefix.

Teacher probabilities describe alternatives at each prefix, not just one preferred next token. Distillation → explains the KL direction. Preference Optimization → explains the sequence margin.

The report gives approximate improvements over SFT of 8% AIME, 10% GPQA, 30% BFCL V4 and 8% preference evaluation. The report calls these relative gains, not percentage-point differences. RL from the distilled checkpoint also improves more than RL started directly from SFT. Component-level evidence is less detailed than the FG-WSD ablation; do not attribute the whole gain to negative-prefix supervision alone.

Refresh reasoning difficulty on-policy

Nanbeige 4 separates STEM, practical code and human-preference stages because capabilities progress at different rates. Its reported on-policy GRPO removes the KL penalty and masks truncated sequences. That is a recipe choice, not a general prescription to remove KL everywhere.

Before each stage, sample sixteen responses per question and retain measured pass rates strictly between 10% and 90%. For binary outcomes, this keeps two through fourteen successes out of sixteen. Resample after the policy changes; “difficulty” belongs to a policy and sampling setup, not permanently to the prompt.

A Python-backed STEM verifier checks numerical/symbolic equivalence. Coding construction can proceed backward from a valid solution to executable public/private tests and then a problem statement. Verifier and prompt must express the same contract; executable tests can still be incomplete or exploitable.

Final Thinking evaluation uses temperature 0.6, top-p 0.95 and maximum output 64K. AIME averages eight runs; most others three. Strong math/function-calling results coexist with weaker full-stack and broader knowledge results. The selected evaluation table below preserves that uneven profile. The Thinking checkpoint is Nanbeige4-3B-Thinking-2511. Broader knowledge, full-stack coding and multi-turn instruction following remain weaker dimensions.

Nanbeige 4.2: reuse depth, not cache state

Nanbeige 4.2 shifts toward repository coding, tool workflows, office artifacts and recovery. It trains a two-pass stack from scratch on 28T tokens. The reported recipe increases math, code, synthetic QA and a small agent-trajectory component; complete mixture weights and compute remain undisclosed.

The checkpoint has 22 physical layers, width 3,072, FFN width 10,752, 48 query heads, 8 KV heads and head width 128. Two passes create 44 layer applications with one stored stack. Maximum configured context is 262,144; this is not an assurance that every task works at that length.

From-scratch looping beats upcycling in the report's experiments. Upcycling starts with an ordinary pretrained model, adds the loop, then continues training. Learning compatible representations from initialization is the authors’ explanation, not an isolated causal result. The authors report about 75% of standard-transformer token efficiency for two passes. They disclose neither a full loop-depth table nor enough conditions to use that figure as a serving-speed prediction.

Separate pass caches give greater quality gains than sharing them. The chosen full loop therefore retains both cache states. Sharing halves this cache component but changes the computation. Architecture and data both change relative to Nanbeige 4, so base improvements do not isolate looping. The 28T/3B ratio is approximately 9,333 training tokens per non-embedding parameter, not a FLOP count. Nanbeige 4.2 report ↗

For the verified query, KV and output projection shapes, see Nanbeige’s attention-width comparison →. The attention inner width need not equal the residual width.

Trace one token through two passes

For 22 layers, two passes and BF16-like two-byte storage, 32K requires 5.5 GiB analytically. At 256K, it requires 44 GiB. Quantization, offload or a different backend can change storage, but parameter reuse alone does not make maximum context practical on a phone.

Each token passes through both depths before readout. Later tokens need the corresponding earlier states for each effective depth. Collapsing pass-indexed caches changes those inputs; it is not simply an allocator optimization. Recurrent and Looped Models →

Build tasks with executable consequences

Repository tasks begin at a reference patch's parent commit. Reconstruct dependencies and tests in a container. Hide the reference patch and grading tests from the agent. Fail-to-pass tests confirm the missing behavior; pass-to-pass tests protect existing functionality. Neither alone guarantees that all relevant behavior is correct.

Mine current model failures for new repository/patch search cues. Generate trajectories through Claude Code, OpenHands, SWE-agent and specialized Codex drivers. Their prompts, editing interfaces and context policies differ. Diversity reduces one-interface overfitting only if those differences remain compatible with the intended tool contract.

Complex tool-use synthesis starts from thousands of Model Context Protocol (MCP) specifications. Tools are bundled for composable search, data access, repository operations, execution and testing. It combines executable local Python tools backed by real data, live services for changing information, and model-simulated tools for expensive runtimes. Simulated realism needs independent checks; a correct simulated task can still fail on a real service.

Generate an instruction together with a Python verifier. Repeated solver attempts guide difficulty evolution along tool-chain depth, retrieval, parameter inference and cross-tool composition. Office synthesis clusters documents, slides, spreadsheets, code and other structured files into coherent bundles. A sandboxed synthesis agent creates tasks, rubrics, trajectories and deliverables. An independent judge checks those outputs before they return to the artifact pool. Recycling validated deliverables expands the pool but can amplify synthetic biases.

Repository trajectories must pass target/regression tests. Turn-level filtering rejects invalid calls, non-terminating loops, redundant actions and context truncation. Keep a disjoint fast validation suite; do not let iterative task generation leak its reference outputs into training.

Preserve recovery without imitating bad turns

The curriculum grows context and agent share together. It first establishes reasoning, then shifts toward long tool interactions. Because both change, the final result does not isolate the effect of context extension.

A successful trajectory may contain a failed action followed by useful recovery. Deleting the failure and observation creates an unreal history. Imitating the failed action teaches the wrong behavior. Keep its context but exclude its assistant target from loss.

Specify token masks and reduction denominators as well as turn labels. A masked target can still influence later loss through its representation unless it is explicitly detached. “No imitation loss on this turn” does not mean “no gradient anywhere through its history.”

Regularize termination before rewarding efficiency

After SFT, Nanbeige 4.2 still repeats, reflects cyclically, duplicates answers and fails to terminate. A point-wise reward model evaluates correctness, relevance, format, termination, safety and user experience. Think-focused RLHF comes first; predominantly Non-Think RLHF follows.

Non-Think training transfers behavioral gains back to Think evaluation. This is not a comparison that switches inference mode to obtain shorter outputs. Accuracy rises from 65.45 to 72.10 while mean output drops from 25,905 to 15,182. The reported bad-case metric reaches zero in this sample, not proof that future failures are impossible.

Behavioral control can expose existing capability by allowing a correct response to finish. It does not establish that reward training added no knowledge or that the improvement comes from one reward component.

Make the length penalty depend on competence

A uniform token penalty may stop hard-problem exploration too early. Nanbeige instead keeps a problem-specific median of correct historical lengths. Penalize only length above that budget, scale the penalty by current success fraction, and alternate constrained phases with free expansion.

At p=0.5 and length 2,500, clipped excess is 0.5 and the penalty is 0.25. Length at or below 1,000 gets no penalty. Length beyond 4,000 reaches the same bounded maximum; this reward expression alone is not an execution stop rule.

The reported budget does not track later improvements: successful histories from earlier checkpoints set it once. Updating that median online would be a different adaptation. Audit missing-history fallback, phase lengths and reward scale before transferring the method.

Agent actions need denser feedback

Terminal success gives weak credit assignment over long tool sequences. Nanbeige combines outcomes with action-centric rubrics for valid calls, information gain, progress and recurring mistakes. Process rewards supplement, not replace, final verification.

For compact agent RL, the recipe favors shorter trajectories and relatively high pass@8. This differs from reasoning's mixed-outcome difficulty filter. Long-horizon action tasks can provide too little useful feedback to a small initial policy. Pass@8 measures at least one success across attempts; it is not the same statistic as the correct-response fraction.

The report shows about a 20% drop in normalized single-turn errors and validation rising from 66.0 to a peak 71.0. The peak is not necessarily the final checkpoint. Full RL improves accuracy and output length across six reported tasks, but reward coefficients, datasets and complete serving cost remain undisclosed.

Gemma: one family, different model designs

Per-layer embeddings explain part of the edge models' stored/effective gap. This is not MoE routing. The shared decoder design uses pre/post RMSNorm, QK normalization and about 262K SentencePiece entries. Encoders and drafters add parameters beyond a simple backbone label.

The 12B encoder-free experiment and 26B MoE are distinct designs, not just larger E2B models. Compare which computation each family member changes before extrapolating a small-model finding. Gemma report ↗

Compose local attention and cache reductions

Gemma uses four local layers per global layer in E2B and five per global elsewhere. Local layers use full RoPE with base 10K. Global layers use rotary fraction $p=0.25$ with base 1M. Periodic full-context access restores communication that a local window cannot supply alone.

Global layers in 12B, 26B-A4B and 31B use keys as values. The unrotated key can supply values; the rotary fraction still needs position-dependent key storage. Under the stated representation, conventional $2d$ storage becomes $d+pd$.

For an illustrative head width of 128 and rotary fraction one quarter, count scalar entries per token. This is a storage calculation, not a measured runtime allocation.

The shared representation and rotated slice total 160 entries. The baseline has 256, so 96 entries are removed. Multiply both payloads by the same element size to compare bytes. Local-layer caches, other layers, quantization metadata and workspaces remain outside this calculation.

E2B/E4B instead report explicit cache sharing ratios $\frac{20}{35}$ and $\frac{18}{42}$. Do not apply the larger models' $K=V$ formula to them. Layer sharing, locality and numerical precision affect different cache terms. Attention → explains the operators separately.

Put modality work in encoders or the decoder

Gemma's frozen 150M/550M vision encoders use 16-pixel patches, variable aspect ratios, axial 2D RoPE and learned absolute coordinates. Image budgets range through 70, 140, 280, 560 and 1,120 tokens. Preserve aspect ratio, round to pooled-patch multiples and choose a representation under the token budget.

The edge audio encoder follows the Universal Speech Model (USM) design and has two downsampling convolutions and twelve Conformer layers. It emits continuous features every 40ms from Mel inputs; no vector quantization is used. Vision and audio encoder weights stay frozen during pretraining. Its 305M size is 55% below the older 680M encoder.

Encoder-free 12B projects 48×48×3 RGB patches through a 35M matrix, adds coordinate-derived embeddings and LayerNorm, then passes them into the decoder. Audio uses raw 16KHz samples in 640-sample/40ms chunks. Representation learning moves into the shared decoder; eliminating an encoder does not eliminate that work.

Competitive audio results establish viability for this model. They do not prove superiority under equal model size or compute. Frozen encoders can supply strong representations cheaply; encoder-free inputs can simplify runtime and fragmentation. Measure end-to-end cost and quality at the same modality budget.

Train the arithmetic and interface you will serve

The caption's “Gb” conflicts with the usual byte interpretation of parameter footprints. These are source values, not silently normalized units. The MoE active weights are 7.6 raw/2.8 quantized, while total is 52/16.2. Active weights must not substitute for resident inventory. E-model effective footprints also do not prove all stored parameters fit in that memory.

Mobile checkpoints use mixed per-channel INT2/INT4 weights and INT8 activations; open-runtime checkpoints use blockwise Q4_0. Per-block scalar bounds separately support FP16 activations. Precision, packing and runtime kernels form one export contract.

Vision W8A8 reports 400→200MB forward memory including compilation overhead. Its 44% latency comparison also changes the older model/hardware. Audio uses layer-cluster weight precisions of 2, 4 and 8 bits with 8-bit activations. Audio disk size 390→87MB compares a redesigned smaller encoder with mixed precision; it is not an isolated quantization ablation.

Gemma's four-layer autoregressive drafter—three local layers and one global layer—uses a separate embedder, previous main activation, token embeddings and cross-attention to main KV. It avoids separate draft prefill and supports variable draft length. For edge models, clustered output reduces the final d×262K projection to d×4,096. Similar reported acceptance does not establish unchanged end-to-end quality without the verifier. Speculative Decoding →

Pretrained termination uses <eos>; instruction turns use <turn|>. A leading-system <|think|> selects reasoning, with the reasoning wrapper <|channel>thought …<channel|>. Tool declarations and calls have separate control tokens. Preserve the checkpoint's template and stop contract during fine-tuning. A successful tensor load does not prove a compatible conversation interface.

Training infrastructure is part of the recipe

Gemma pretraining includes web, code, images and audio for E2B/E4B/12B, with a January 2025 cutoff. Benchmark contamination, unsafe material and recitation risk are filtered. Token counts, mixture weights, optimizers and detailed curricula are not disclosed.

Training uses TPUv5p/v6e, 4,096–12,288 chips, with data/sequence/replica partitioning, ZeRO-3 optimizer-state sharding, Pathways reductions and JAX/GSPMD/XLA. Slice-Granularity Elasticity continues with fewer slices after localized failure. Reported interruption falls from minutes to seconds; temporary lower capacity can improve goodput, not peak FLOPs.

Post-training adds thinking behavior and filters personal information, toxicity, mistaken model identity and duplicates. Attribution, calibrated hedging and refusals teach how to respond when evidence is weak. Reported factuality gains do not disclose enough data or reward detail to reproduce the pipeline fully.

E2B approaches older 27B text results on selected tasks; E4B approaches its vision results with a small InfographicVQA deficit. For 31B at 128K, RULER is 96.4 versus 66.0 and LOFT recall 79.5 versus 8.6. E4B also improves reported average speech translation and transcription over Gemma 3n with the smaller quantized audio encoder. Most static Gemma 4 results use thinking mode while Gemma 3 often does not. Vision results at 1,120 versus 280 tokens are not interchangeable. A dated Arena rank is product evidence, not a stable architecture property.

Apple: choose a flash-backed subnetwork

Apple Foundation Models (AFM) became a five-model stack in the June 8, 2026 overview. The tiers separate local latency and memory constraints from cloud reasoning and image generation. Private Cloud Compute (PCC) is Apple's server-side deployment environment, not a sixth model.

Core Advanced addresses flash-to-DRAM bandwidth. A standard resident MoE can route every token cheaply relative to its memory bandwidth. The same policy may become too expensive if every new expert choice requires loading weights from NAND flash. Low active parameter count alone does not solve that transfer cost.

1. Store the inventory. Keep the full sparse model in NAND; retain always-active shared weights.
2. Select for the request. A lightweight dense block chooses a fixed subset of routed experts during initial processing.
3. Load a working subnetwork. Move selected experts into DRAM and combine them with shared weights.
4. Generate and reselect. Run that subnetwork for a period, then periodically replace the active expert set.

Apple calls this Instruction-Following Pruning (IFP). The request receives a predetermined active-parameter budget based on its use case. This is inference-time elasticity, not token-by-token top-k routing with all experts already resident. The overview does not disclose a complete expert layout or reselection interval. MoE routing → separates these choices. Apple overview ↗

Specialize a common foundation for each tier

The overview reports scaled pretraining on cloud TPUs, followed by architecture/use-case specialization. The common initial foundation branches toward audio, image understanding, long-context reasoning and image generation. Supervised fine-tuning and multi-stage RL follow; quantization-aware training targets the deployment hardware.

AFM 3 Cloud builds on the 2025 Parallel-Track Mixture-of-Experts (PT-MoE) foundation. Apple attributes greater training stability, reasoning and context recall to architectural refinements. Those broad claims do not disclose the 2026 layer layout. On-device IFP and cloud PT-MoE are different designs.

Token counts, optimizers, training context lengths, full sparse layouts and scaling regimes remain undisclosed in this overview. The June announcement's promised summer report is a historical statement, not evidence about the current existence of every later report. 2025 report for historical context ↗

The overview's September 9 update adds an opt-in exception for improvement data. Do not repeat an unconditional claim that user interactions are never used. This describes the stated data policy; it is not an independent privacy audit.

Route the application, not just its tokens

The Foundation Models framework exposes on-device and PCC models through one Swift API. The June developer session reports image input on-device and common structured-output/tool interfaces. The model selector changes where computation occurs; it does not guarantee identical quality or behavior.

Use local inference when offline availability, latency and local processing fit the task. Escalate to PCC when the task needs more context or reasoning. Evaluate the actual feature and handle server unavailability or usage limits. A unified API makes switching easier; it does not remove those deployment differences. WWDC26 session319 ↗

Read Apple’s product comparisons in scope

The 2026 overview emphasizes in-house human preference, feature-specific quality and comparisons with the 2025 generation. These are useful product signals, not a complete standardized cross-model benchmark suite.

Core Advanced's sparse local execution, integrated audio/multimodality and the explicit cloud/image/reasoning tiers distinguish this stack from the simpler earlier local/server split. The evaluation does not isolate flash-backed routing as the cause of every quality gain. Nor does a tier's deployment target establish its complete training recipe.

Compare scores with their complete budgets

Nanbeige 4.2 evaluates at 256K context. Code-agent runs allow 250 turns, outputs up to 32K and four-to-ten-hour timeouts; most code-agent results average eight runs. Its reported SWE Verified 63.6, SWE Pro 46.9 and Terminal Bench 44.1 belong to that model–scaffold–tools–budget combination.

Nanbeige leads these reported general-agent and code-agent comparisons. It does not win every reasoning and alignment task: Gemma leads SciCode, IF-Bench and Recruit-Bench. Current weights, retry policy, context compression and model-judge reliability all affect agent scores. Match those conditions before attributing a gain to looping or parameter size.

For Nanbeige 4, quality-labeling models, retrieval infrastructure and many optimization details remain unavailable; some teachers and evaluators are proprietary. Final comparisons do not isolate component effects. Raw/filtered datasets, teachers, reward systems, task generators and much training compute remain unavailable. Some tests use in-house harnesses or benchmarks; several agent tests use model judges. Matched settings support within-harness comparisons, not scaffold-independent intrinsic model rankings. Preserve uncertainty at the component level even when final quality is strong.

Turn a recipe into a controlled experiment

First name the bottleneck: vocabulary storage, per-token depth, long-context cache, malformed actions, sparse verification or expert-transfer traffic. Change one relevant mechanism under a matched budget and retain its supporting conditions.

For looping, verify pass-indexed caches and compare matched data/compute—not only stored weights. For masking, inspect target spans and denominator while retaining causal observations. For length control, test missing history, zero success and denominator limits. For deployment, validate the exact dtype, quantization packing, template, hardware and modality workload.

Training Data →, Scaling Laws → and Precision → own the reusable mechanisms. WSD scheduling →, SFT → and RL → connect the reusable training choices. The complete recipes here explain why those choices appear together without treating one final benchmark as an ablation.

Sources and connected recipes

liquid ↗ · nan4 ↗ · nan42 ↗ · config ↗ · gemma ↗ · apple ↗

Nanbeige checkpoints ↗ · Positional Encodings → · Serving Models → · Run Design and Goodput →

Parameter, cache and reward demonstrations are analytical examples. Measured behavior and schedule tables retain the report conditions. Missing protocols are not filled with guessed defaults.