Building and Controlling Reasoning Models

Reasoning is a system, not a token count

A reasoning system spends computation to produce, inspect and choose an answer. The model supplies proposals. A generation policy decides how many proposals to produce. A verifier or selection rule decides what evidence counts. The evaluator measures whether the whole procedure works.

Here a reasoning model operationally generates intermediate problem-solving tokens before a final answer. Those tokens can support decomposition, checking and repair. They do not establish that every written step caused the answer, or that the model reasons like a human.

This note follows one implementation chain: measure reliably, improve inference, update the policy, transfer useful behavior, then control its cost. Each stage depends on the previous contract. More sophisticated training cannot compensate for a verifier that consistently rewards the wrong result.

What building from scratch actually builds

Sebastian Raschka's book starts from pretrained open Qwen3 weights. It does not pretrain a model from random initialization. Building from scratch means implementing the reasoning machinery directly instead of hiding it behind a post-training framework.

The implementation includes autoregressive generation, key/value caching, answer extraction, sampling, scoring, refinement, rollouts, rewards, group-relative advantages, policy gradients, stabilization and teacher-trace distillation. The 0.6B model makes these dependencies inspectable on modest hardware.

Pretraining supplies knowledge and basic operations. Inference procedures elicit and select behavior. Post-training changes the distribution of that behavior. These roles overlap in their effects, but they have different data, compute and evaluation contracts. Build a Reasoning Model (From Scratch) ↗ and its companion implementation ↗ provide the worked context.

Reuse the same primitives across the pipeline

Answer extraction first measures accuracy, then supplies a reward and filters teacher examples. Token log probabilities first score responses, then enter a policy loss. Reusing code is useful only when the contract remains explicit: an inference score is not automatically a calibrated verifier.

Build and test each operation independently. A change to the parser can change reward without changing the model. A template change can change generation without changing the optimizer. Record both in experiment provenance.

Freeze the measurement before improving the model

Establish a reproducible greedy baseline before introducing sampling or training. Freeze prompts, answer rules, token limits, checkpoint, seed and decoding settings. Store the raw response alongside the extracted answer and the correctness decision.

The book uses MATH-500 as a held-out benchmark and non-overlapping MATH problems for training and distillation. Split by underlying problem, not just row or response. Several traces from one question are not independent evaluation questions.

Repeated checkpoint selection can overfit a validation set even without gradient updates on that set. Keep a final test set outside model and hyperparameter selection. Report extraction failures separately from mathematical errors. Otherwise a cleaner wrapper can look like improved reasoning.

Extract the final answer without guessing its meaning

Prefer an explicit final-answer marker, such as a LaTeX boxed expression. This reference selects the last literal boxed marker. If that marker has no balanced ending, extraction fails rather than falling back to an earlier answer. Nested expressions require balanced braces; stopping at the first closing brace can truncate a fraction.

def extract_last_boxed(response):
    marker = r"\boxed{"
    start = response.rfind(marker)
    if start < 0: return None
    depth, escaped = 1, False
    for end in range(start + len(marker), len(response)):
        char = response[end]
        if escaped: escaped = False; continue
        if char == "\\": escaped = True; continue
        depth += (char == "{") - (char == "}")
        if depth == 0: return response[start + len(marker):end]
    return None

This original Python reference extracts a payload; it is not a complete LaTeX parser or a verbatim book listing. It skips escaped braces and recognizes the literal marker shown in the function. It does not interpret comments, macros or whether a marker occurs in a quoted example. Test those cases separately before using another output grammar.

No reference answer enters this parser. A fallback that takes the last number can extract a step index, denominator or abandoned intermediate result. If a benchmark permits a fallback, log which rule produced the answer. Parser failure should remain visible instead of being repaired with knowledge of the expected answer.

Compare meaning after normalizing the surface

Normalize superficial differences first: permitted whitespace, wrapper notation and benchmark-specific formatting. Then test exact equality. When strings differ, a symbolic checker can simplify their difference and test whether the result is zero.

Surface normalization is not mathematical equivalence. The answers x and −x differ. An equation can have domain restrictions. Sets, units, intervals, root branches and ordered tuples need their own comparison rules. Do not strip signs or reorder every list merely to increase apparent accuracy.

Treat generated expressions as untrusted input. Use restricted parsing, timeouts and isolated symbolic execution where appropriate. Never execute generated code through an unrestricted evaluator. Test valid equivalent expressions, valid unequal expressions, malformed syntax and resource-intensive cases.

An evaluator is not always a deployment oracle

On a verifiable deployment task, tests or an exact calculation may select among answers. On an ordinary unknown-answer question, a learned judge or voting rule supplies a proxy. Do not report oracle candidate coverage as deployable selection accuracy.

The same bug can affect measurement, reward and teacher filtering. Agreement between these three stages is not independent confirmation when all use one faulty checker. Include hand-reviewed failures and independent tests for the verifier itself.

Give one trajectory more room

A step-by-step prompt gives a single generation more token positions in which to transform its state. The model can decompose a problem, carry intermediate quantities and reconsider an operation. This changes the generated trajectory without changing weights.

The cost is sequential generation, longer context and a larger KV cache. A long trace can repeat the same error or introduce a new one. The visible number of words is only a proxy for useful computation. Compare final accuracy and latency, not length alone.

Keep final-answer tokens distinct from reasoning tokens when the serving interface supports that split. A system can end reasoning yet still need enough tokens to state the result. A total-output cap and a reasoning-only cap therefore have different failure modes.

Generate alternatives, not copies

Parallel sampling explores several trajectories from the same prompt. Temperature and nucleus sampling change which alternatives are likely. The book suggests temperature 0.5–0.9 and top-p 0.7–0.9 as starting ranges, not universal optima.

Too little diversity produces copies. Too much randomness can destroy useful structure. Fix the decoding contract while comparing sample counts, then sweep diversity separately. Decoding explains the logit transformation and support changes.

Different seeds do not ensure independent reasoning errors. Several trajectories can share the same missing fact or attractive wrong shortcut. Answer diversity and trajectory diversity are related but not identical: distinct explanations can reach the same answer.

Select an answer through self-consistency

Self-consistency generates N solutions, extracts their final answers, groups equivalent results and selects the most frequent group. The explanations need not use the same path. Grouping must use the same domain-aware equality contract as evaluation.

Many implementations call this majority voting even when the winner has less than half the votes. That is plurality selection. State the tie rule: fixed ordering, another sample or random choice. Using the known correct answer to resolve ties turns the result into an oracle experiment.

answers = [extract(sample(prompt)) for _ in range(N)]
groups = group_equivalent(answers, comparison_contract)
counts = count_valid(groups)
return choose_plurality(counts, declared_tie_rule)

Also declare how invalid answers contribute. Dropping parser failures changes the effective sample count. Counting them as an answer can make failure the largest group. Neither policy should be hidden.

Separate having a solution from selecting it

This probability describes candidate coverage, not the probability that a selector returns the correct candidate. An oracle identifies any success. Voting instead depends on how wrong answers cluster. A scorer depends on whether its ranking tracks correctness.

The example fixes per-draw correctness at 40%. Splitting wrong answers helps plurality because no single wrong group receives all the remaining mass. Sharing a draw erases independence. The shared-draw parameter is a mixture probability, not a general correlation coefficient.

For benchmark estimates of pass@N, distinguish the observed fraction of successful groups from a sampling estimator based on a larger pool. Keep sample dependence, seed counts and task averaging explicit.

Stop only when unseen votes cannot matter

Suppose one answer leads the next answer by m votes and r samples remain. If m > r, no unseen assignment can change or tie the leader. Stopping at that point preserves the final plurality decision for the fixed sample budget.

The strict inequality is conservative. If m = r, the runner-up can still tie; the declared tie rule matters. At the final sample there is no work left, even when counts tie. Stopping because the leader looks convincing has a different statistical contract.

Adaptive stopping may save generations, but it also changes observed runtime and the dataset of completed candidates. Record the stopping rule and compare it against the same maximum budget. A fixed-budget proof does not justify arbitrary confidence-based stopping.

Score outcomes or partial solution states

A proposer generates candidates. An outcome reward model (ORM) scores a completed answer. A process reward model (PRM) scores intermediate states or steps, which allows pruning before full solutions finish.

A PRM score need not mean that a step is logically valid. In the Snell et al. study, Monte Carlo continuation outcomes train a reward-to-go estimate. A state can receive a useful score because continuations often succeed, not because every sentence constitutes a proof.

Full-solution selection can use a final-step score or aggregate step scores. These are different estimators. Multiplying scores can punish long solutions; taking a minimum emphasizes one weak step. Choose and validate the aggregation rather than treating all PRMs as the same verifier.

When several candidates reach the same answer, a weighted answer vote can sum their scores. This combines proposal frequency with learned confidence. It is still a proxy unless the scoring evidence proves the result.

Search can exploit the scorer instead of solving the task

Step-wise beam search expands partial solutions, scores them and keeps a bounded frontier. Lookahead additionally generates future continuations before choosing which state to expand. Sequential revisions instead condition a fresh answer on earlier attempts.

Search gives a scorer repeated opportunities to prefer its own systematic mistake. As the search budget grows, the selected candidate may exploit a high proxy score without becoming more correct. More scoring is not automatically more reliable verification.

Count all computation: proposal tokens, discarded branches, lookahead continuations, verifier calls and context replay. A wider batch can reduce wall-clock delay while increasing work. In the Snell setup, k-step lookahead uses a budget proportional to N(k+1), not just N.

Keep a held-out check independent of the search score. Track selected correctness against proxy score as budget increases. Rising score with falling accuracy is a warning about verifier over-optimization, not evidence that the model needs an even wider beam.

Repair an answer without assuming improvement

Self-refinement starts with an answer, asks for a critique and repair plan, generates a replacement, optionally accepts that replacement, then repeats. The critique must identify something repairable; generic requests to reconsider can produce an equally wrong paraphrase.

The blue line tracks the current answer. The burgundy line tracks whether any answer has been correct, as if an oracle retained it. An actual scorer cannot assume that oracle. With enough damage, additional rounds reduce current correctness despite improving historical coverage.

Keep answer history and acceptance decisions. Compare last-answer selection, fixed heuristic gating and an independent verifier. Do not discard the initial correct answer merely because a later response sounds more deliberate. The probabilities above are a constructed Markov model, not a measured revision curve.

Fluency is not a correctness score

Sequence log probability penalizes length through additional negative terms. Averaging removes that direct sum effect, but it does not make scores calibrated across tasks or styles. A familiar wrong answer can have a higher average score than an unfamiliar correct derivation.

Teacher-forced scoring needs the exact token IDs and history used for the response. Exclude prompt positions when the score is response-only. Record whether probabilities come from raw model logits or a temperature/top-p-transformed sampler; these are not the same distribution.

Heuristic scorers can reward formatting, brevity or explicit checking. Those features are useful interface signals, not guarantees of truth. A judge also needs held-out calibration and resistance to persuasive but invalid explanations.

What the book's inference experiments actually measured

The following observations come from Tables 4.1 and 5.1 of Raschka's book. The experiments use Qwen3 0.6B base/reasoning models and all 500 MATH-500 questions on DGX Spark with CUDA. Sampling in Table 4.1 uses temperature 0.9 and top-p 0.9.

Time is total benchmark evaluation time in this implementation, not per-request latency or a general FLOPs estimate. The 52.0% base-model vote-10 result costs 862.6 minutes. Greedy base achieves 15.2% in 10.1 minutes; step-by-step prompting achieves 40.6% in 84.5 minutes.

Without step-by-step prompting, base voting at N=3,5,10 gives 29.6%, 27.8%, 31.6%. Sampling more does not produce a monotonic observed accuracy curve. Distinguish finite-benchmark variation from a universal statement about the population curve.

For the reasoning model, one unscored revision reaches 56.6%; heuristic gating reaches 57.8%; average-log-probability gating reaches 48.4%. All take about 499 minutes. These results show why confidence gating must be tested, not assumed beneficial.

The tables are observations from a small-model math experiment. They do not establish rankings for every model, task, implementation or hardware configuration. Book, chapters 4–5 ↗ contains the surrounding procedure.

Difficulty changes the useful compute allocation

Snell et al. study compute allocation using PaLM 2-S* (Codey), 12,000 MATH training problems and 500 test problems. They train task-specific verifiers and revision behavior. Their conclusions are not results for an arbitrary unchanged chat model.

The study estimates difficulty using 2,048 samples per problem and divides questions into five bins. Strategy selection uses two-fold cross-validation. Difficulty-estimation overhead is mostly outside the reported generation-budget comparisons, so a practical router must account for that extra work.

For PRM search, beam search can help intermediate difficulty bins but over-optimize easy questions as budget grows. The hardest bin gains little. Lookahead often underperforms at matched generation budgets. This does not support a universal rule that easy means sampling and hard means beam search.

For revisions, easy questions can benefit from sequential repair; harder questions favor a mixture of parallel proposals and sequential revisions. Training supplies up to four prior answers, with older history truncated when more attempts occur. Naive revision changes about 38% of initially correct answers into incorrect ones in that setting.

The study reports up to 4× fewer generation-budget units than best-of-N in selected comparisons. That is not a 4× end-to-end latency claim. Scaling LLM Test-Time Compute Optimally, main §§3–8 ↗ separates search and revision experiments.

Inference and pretraining do not have a fixed exchange rate

Extra inference is paid per deployment request. Extra pretraining is paid once and reused across requests. Whether one replaces the other depends on deployment load, task difficulty and the model's ability to use additional computation.

Derive the expression by equating 6MNDpre + 2MNDinfer to 6NDpre + 2NKDinfer. It excludes many implementation costs and does not predict quality. Small R makes the one-time pretraining difference large relative to the deployment token budget.

The study includes a roughly 14× parameter-size comparison under its FLOPs accounting. It does not establish a fixed exchange such as 2× inference replacing 10× pretraining. A useful comparison needs a matched total-compute boundary and the actual request distribution.

See Scaling for pretraining allocation. Serving must additionally measure parallelism, queueing and verifier cost; equal FLOPs do not imply equal latency.

Change the policy with verifiable rewards

Inference allocation leaves model weights fixed. Reinforcement learning with verifiable rewards (RLVR) changes those weights using checked outcomes. Math answers, executable tests and environment states provide the reward evidence.

Group Relative Policy Optimization (GRPO) uses relative rewards rather than a separately trained value critic in this basic form. RLVR names the reward source; GRPO names an update method. RLVR can use other policy-gradient methods.

If all rewards equal the same value, every advantage is zero. Binary all-pass and all-fail groups provide no relative correctness signal. Mixed-success tasks can provide useful contrast, but choosing only the hardest tasks can eliminate successes. Curriculum and reward granularity change this tradeoff.

Additional format or process rewards can create variation even when correctness is uniform. That variation teaches the additional criterion, not new correctness evidence. Reinforcement Learning owns the detailed estimator and normalization choices.

Build the minimal online update in the right order

For each prompt, generate a group using a recorded behavior checkpoint. Extract the answer and compute reward. Freeze those rewards and advantages. Recompute response token log probabilities with gradients enabled, then form and backpropagate the update loss.

responses, behavior_logp = rollout(policy, prompt, group_size)
rewards = verify(extract(responses), reference)
advantages = group_normalize(rewards).detach()
current_logp = score_with_grad(policy, prompt, responses)
sequence_logp = (current_logp * response_mask).sum(-1)
loss = -(advantages * sequence_logp).mean()
loss.backward()
optimizer.step()

This is conceptual pseudocode for the minimal sequence-summed update. It omits optimizer zeroing, accumulation, clipping and several production contracts. Sample generation normally runs without a gradient graph; the later scoring pass supplies differentiable probabilities.

A causal language model predicts each response token from preceding context. Shift logits and targets consistently. The response mask excludes prompt and padding positions; include or exclude end markers by explicit policy. A malformed mask can train the prompt or erase all useful feedback.

Understand what the short policy loss omits

Summing tokens weights longer responses differently from averaging each response. Both can be implemented correctly, but they define different objectives. State the loss reduction and mask denominator. Do not call every advantage-weighted cross-entropy a complete stable GRPO system.

Repeated updates on stored rollouts make the current policy differ from the behavior policy. Importance ratios and clipping address that mismatch under their assumptions. A separate reference policy provides a KL anchor. Behavior and reference are different roles even if they initially share weights.

Large asynchronous training additionally needs collection versions, exact sampling probabilities, stale-trajectory handling and distributed numerical checks. The short kernel exposes the gradient mechanism; it does not replace those contracts.

Fifty steps are evidence, not a universal schedule

Evaluation allows up to 2,048 generated tokens. The training maxima of 256 or 512 tokens are not the evaluation maxima. The 512-token, eight-response run reaches 47.4%, compared with the 15.2% greedy base baseline.

Mean evaluation lengths are 560.22, 579.81 and 586.11 tokens for the three runs. The base and official reasoning references average 78.85 and 1,369.79 tokens in this comparison. Similar accuracy does not mean identical behavior or identical cost.

Longer training can reduce held-out accuracy. Save early checkpoints and evaluate the trajectory rather than assuming the last update is best. One small-model math experiment does not establish that fifty steps suffice for arbitrary capabilities. Book, chapter 6 ↗ supplies the educational setup.

A useful objective can still produce an unstable run

Track held-out accuracy alongside training reward. A growing reward can reflect exploitation of extraction rules, a narrow curriculum or a more verbose style. Track response length, entropy, reward variation, advantages, losses and policy ratios.

Clipped ratios limit a surrogate update, not every possible policy change. KL discourages drift from a reference, but can also constrain useful learning. Format rewards improve parsing, not mathematics. In the book's small experiment, adding KL did not prevent eventual collapse.

These controls interact with learning rate, rollout group size, response length, reward design and prompt difficulty. Change one declared factor at a time when diagnosing failure. Training Stability adds run-level diagnosis; RL explains the update controls.

Make correctness observable through a reliable interface

A verifier can reward only what it can identify. Reasoning delimiters and a predictable final-answer field make extraction and runtime parsing reliable. They are interface conventions, not evidence that the content inside is valid.

Separate outcome reward from format reward. Correctly wrapping an incorrect answer should not outweigh a correct answer expressed through a recoverable interface. Conversely, deployment may require strict schemas, so correctness and schema compliance both need evaluation.

Test missing delimiters, extra answers, an unfinished reasoning block and an early end-of-sequence token. Decide whether malformed output earns zero, a partial format score or a separate diagnostic. Keep that rule stable across checkpoints.

Turn teacher computation into reusable behavior

Hard reasoning distillation pays the teacher generation cost while collecting data. Generate traces on training prompts, verify their final answers, filter unsuitable records, apply the student's template and train on the teacher's response tokens.

This supplies dense supervised targets instead of a scalar reward for an online student rollout. It can establish useful trajectories when the student rarely earns any reward. It can also copy teacher errors, unnecessary verbosity and a narrow response style.

Keep prompt-family splits, source checkpoint, generation settings and filtering decisions. Verify the answer without claiming that every intermediate sentence is correct. Mask prompt/padding positions as intended; make the final answer and reasoning targets explicit.

A useful sequence is teacher SFT, student RLVR, then calibrated inference allocation. It is not mandatory for every model. Distillation separates fixed traces, full-distribution targets and on-policy teacher feedback.

Teacher strength does not determine student transfer alone

Qwen3 235B-A22B traces produce 45.0% after one epoch, then 43.8% and 44.2%. DeepSeek-R1 traces produce 30.6%, 32.4% and 33.6%. A decreasing validation loss does not always predict increasing answer accuracy.

The distillation evaluation example allows up to 4,096 tokens. Do not silently treat that budget as matched to every earlier inference or RL experiment. The 48.2% official reasoning reference is useful context, not a controlled claim that these methods have equal cost.

Raschka suggests that family alignment may help transfer through tokenizer, prompting conventions and response style. The teacher comparison does not isolate that cause. Teacher strength, traces, data and formatting can also differ. State family alignment as an author hypothesis, not a theorem.

A small student cannot necessarily reproduce every teacher capability. Evaluate new problems and long contexts, not only training-like math. Book, chapter 8 ↗ contains the source observations.

Choose which compute bill to pay

Account for three bills separately: teacher/data generation, training, and recurring inference. A cheaper student can amortize teacher cost over many requests. A one-off hard question may justify expensive search without any training.

First sweep reproducible inference baselines. Then compare training interventions against the best relevant baseline, with token and runtime accounting. A trained model that beats greedy decoding may still be less useful than an existing inference-only configuration.

Train correctness and length behavior together

A hybrid checkpoint supports concise answers and extended reasoning. The model learns what to say and how much intermediate computation to express. The two modes can have very different target lengths and stopping behavior.

Explicit examples teach each mode. Evaluation must measure each mode separately: task accuracy, reasoning length, answer length, schema compliance and latency. An average across modes can hide the loss of the concise interface.

If task reward consistently favors long traces, RL can make the nominal concise mode verbose. Conditional costs or distinct mode-specific training can counter that pressure. Separate instruct and reasoning checkpoints simplify some operations, but maintaining both is also a cost. Neither arrangement is universally preferable.

The Smol Playbook discusses the practical interference between concise and reasoning behavior. Staged run design retains that context; Smol Training Playbook ↗ is the source.

Make the mode part of the runtime contract

/think     → <think>{reasoning}</think>{answer}
/no_think  → <think></think>{answer}

These are illustrative conventions, not commands that every model understands. The model must learn the markers. The runtime must parse them consistently and distinguish intermediate output from the delivered answer.

Declare whether earlier reasoning remains in multi-turn context. Stripping it changes later conditioning; retaining it changes token use and may preserve useful or misleading state. Match training and deployment where possible.

An empty prefilled block can suppress the visible reasoning channel. That does not prove that the underlying network performs no internal computation. Likewise, a visible block does not prove faithful introspection. SFT and Adaptation owns the general template and target contract.

Count both examples and supervised token mass

Counting examples is insufficient when one concise response has 100 targets and one long trace has 2,000. The second contributes twenty times as many targets under a global token-average objective. Equal example counts need not mean equal optimization exposure.

However, token count alone is not the complete answer. A per-sequence average gives each sequence a different weight from a global token average. Mode weights, sampling, padding masks and loss reduction all determine effective contribution.

Log examples, supervised token mass and effective loss weights for each mode. Evaluate both behaviors after SFT and again after RL. Do not declare the dataset balanced merely because its row counts match.

Teach the control before trusting the label

An effort label asks for an operating point. Effort-conditioned SFT pairs labels with suitable traces. Effort-conditioned RL can make token cost depend on the label. A model does not reliably acquire these meanings from an arbitrary system instruction alone.

A higher token cost can prefer a shorter candidate even when its success probability is lower. A lower cost can justify more computation. The four candidates are fixed constructed points; moving the slider does not train a policy or establish a measured frontier.

SFT can establish recognizable modes, while RL calibrates success and cost within them. Continuous controls require evaluation between training points, not just at named endpoints. Never invent undisclosed coefficients from a model's user-facing labels.

A learned effort mode is not a hard ceiling

A learned mode describes how to spend available budget. A hard limit describes how much budget is available. Combine them only after testing the interruption contract. Cutting a derivation halfway through can produce an incomplete or unsupported answer.

Reserve final-answer capacity independently when required. Also count prompt replay, tool output, verifier calls and retry generations. A reasoning-token ceiling alone does not bound end-to-end work.

Keep an interrupted trace's answer and declare its mask

Budget-aware data can truncate a reasoning trace at several points while retaining its original final answer. An externally inserted closing marker tells the runtime that the answer begins. Mask that inserted marker if it is context rather than a desired generated action.

The retained reasoning and answer can remain supervised targets. The inserted closure remains conditioning. At a naturally completed trace, the model-generated closure can be supervised. These are distinct masks, even when the visible string is identical.

Such examples train the answer distribution after interruption. They do not guarantee successful finishing after every possible cutoff. Evaluate random and adversarial interruption points, including zero reasoning budget, incomplete calculations and traces close to natural completion.

Fusion and continuous effort conditioning

Qwen3 combines long-chain-of-thought SFT, reasoning RL, thinking-mode-fusion SFT and general RL. Fusion brings thinking and non-thinking examples into one checkpoint and template. The reported inference budget can externally stop the thinking span.

The effort overview describes Qwen3's partial-trace handling as emergent rather than explicitly trained through that exact interruption procedure. Do not attribute another model's masked-truncation recipe to Qwen merely because both expose a budget.

Inkling uses a continuous effort value in the system message and effort-dependent per-token cost during large-scale RL. The public description does not supply every coefficient or full objective. The conceptual cost equation above explains the pattern, not a recovered private implementation.

Qwen3 technical report ↗ · Inkling release description ↗ · Raschka's effort synthesis ↗

Develop effort specialists before consolidating them

DeepSeek V4 develops Non-think, Think High and Think Max specialists with mode-specific post-training settings, context and length costs. Consolidation distills their behaviors into one checkpoint. The report does not disclose every domain-to-teacher mapping.

Kimi K3 similarly consolidates nine domain/effort specialists. The effort synthesis describes developing maximum-effort specialists first, then reducing budgets for high and low settings. Nine teachers does not imply every Cartesian combination of domains and effort levels.

For K3's simple verifiable reasoning example, budget overrun receives −1, while correct and incorrect responses receive 1 and 0 under the stated rule. Do not apply that formula to all agentic or general tasks. Multi-teacher on-policy distillation is a separate transfer mechanism.

GLM-5 describes interleaved thinking, preserved thinking and turn-level thinking. These are three context/interaction behaviors, not low/medium/high effort levels. Templates and subsequent distillation stages make the distinction operational.

Frontier Recipes retains architecture and full training context. DeepSeek V4 report ↗ · Kimi K3 report ↗; these recipe summaries follow the scoped effort synthesis, not an independent full-report replication.

Medium traces and random reasoning budgets

Nemotron 3 Ultra introduces medium-effort teacher traces in SFT, using GPT-OSS-120B as the teacher in the described recipe. About 2.5% of RL prompts receive length-aware control in math, STEM and coding. Regular, medium and off modes coexist with external budgets.

Randomly truncated reasoning traces retain the final answer. An externally inserted closing marker is masked from loss, while it remains available as context. The interruption scene above isolates that mask distinction using invented lengths, not a reproduction of Nemotron's data.

A learned medium mode and a hard reasoning budget can be used together, but they still need separate evaluations. Measure compliance, finishing accuracy and cost at each operating point. Nemotron 3 Ultra report ↗ is the primary source; the effort overview ↗ summarizes this recipe.

Alternate constrained and free training phases

Kimi K2.5's Toggle alternates constrained and unconstrained RL periods. The constrained period encourages useful work within a problem-specific budget. The free period preserves the ability to use additional computation when it helps.

The description uses lengths of correct responses to determine a budget percentile and an accuracy condition to regulate pressure. This is a training schedule, not an inference algorithm that automatically chooses each request's effort.

The report describes roughly 25–30% token reduction against K2 Thinking under its comparison conditions. The number is not a universal reduction for every model or task. A lower average length must be paired with retained capability and the matched evaluation setup.

Alternating phases is coherent because permanent short-budget optimization can teach the model never to explore a longer trajectory. Evaluate constrained performance and unconstrained scalability separately. Kimi K2.5 report ↗ · effort-control synthesis ↗

Increase length pressure as a problem becomes solvable

Nanbeige4.2 increases the penalty when a response exceeds the historical budget and the current policy solves the problem reliably. Responses within budget receive no excess-length penalty. Low current pass rate reduces pressure, leaving more room for exploration.

Historical difficulty and current competence play different roles. Recomputing the budget from the same responses would change the objective. Keep the checkpoint history used for bq separate from the group that supplies pq.

The recipe alternates constrained phases with free-expansion phases where the length penalty is disabled. It therefore does not equate shorter with better under every condition. Compact Model Recipes preserves the broader Nanbeige pipeline; Nanbeige4.2 report ↗ provides the source.

More effort can stop helping or introduce errors

More generation can fail because the model lacks a fact, repeats a strategy, damages an already correct answer or optimizes verbosity. A hard task may need a new tool, a better proposal model or new training evidence rather than another thousand tokens.

If a fixed proposal has exactly zero probability of producing a correct candidate, independent resampling cannot produce one. That statement does not cover a procedure that changes the proposal through feedback, tools or training. Softmax positivity also does not imply practical coverage of a very rare long sequence.

Quality can saturate or decline with effort. Inspect failure families and length buckets, not only aggregate accuracy. Compare more samples, longer traces and stronger verification separately: these spend computation on different bottlenecks.

Compare operating points, not provider labels

Serving has two independent knobs: choose a different model, or change the inference procedure for the same weights. A smaller model at high effort may match a larger model at low effort on some tasks. That is a task-conditioned operating point, not a universal equivalence.

Provider labels are not a common compute unit. High effort in one model need not equal high effort in another. Token prices are not FLOPs; FLOPs are not wall-clock time. Compare configurations on the same workload and constraints.

For throughput-sensitive service, parallel candidates can increase memory pressure and reduce admission capacity. A faster isolated request can still worsen queueing for other requests. Serving owns the scheduler and KV-cache lifecycle.

Escalate from evidence, not confidence alone

Begin with a cheap configuration where errors are inexpensive and easy to detect. Escalate after evidence such as a failed test, an unresolved consistency check or a missing required artifact. Keep enough remaining budget for verification and delivery.

Classification, extraction and routine rewrites can be good low-effort candidates, but do not route only by task name. An ambiguous extraction with high error cost differs from a routine one. Difficult coding or planning may benefit from more compute, but a missing external fact may not.

An automatic router estimates benefit before it knows the answer. That estimate has its own inference and calibration cost. A model's verbal confidence is not sufficient evidence of correctness. Include a user override for latency, cost and reliability priorities.

An agent can change effort across turns when its harness preserves the contract. Record the decision, failed check and remaining budget. Agents and Environments explains why task success depends on executable evidence, not just a persuasive final response.

A compact implementation and evaluation checklist

Compare seeds or confidence intervals where feasible. MATH-500 accuracy changes in increments of 0.2 percentage points for one complete evaluation. A one-seed improvement can be real in that run without establishing a robust population ranking.

The book is an educational reference, not a complete large-scale asynchronous recipe. Its small model and verifiable math domain expose mechanics clearly. Transfer to open-ended tasks requires a stronger evaluation contract, not just a larger batch.

How the branches connect over time

This is a conceptual development map, not a claim that each branch replaced its predecessor. Sampling, RL and distillation can coexist in one deployed system. New reports usually refine one stage of the same pipeline.

Sources and connected topics

Raschka: Build a Reasoning Model (From Scratch), Manning 2026 ↗ · Companion code ↗ · Snell et al.: Scaling LLM Test-Time Compute Optimally ↗ · Raschka: Controlling Reasoning Effort in LLMs ↗

Book tables above reproduce numerical observations with their experimental context, not licensed page images. The five interactive explanations are constructed models. They are not empirical training trajectories, measured provider curves or private training objectives.

RL · Distillation · SFT and Adaptation · Agents · Decoding · Scaling · Frontier Recipes · Compact Recipes