Distillation

What should the student inherit?

Distillation trains a student from a teacher's behavior. The teacher may supply a label, an answer, a probability distribution or a preference. The choice determines what the student can learn. A sampled answer transfers one path; a distribution also describes alternatives.

Compression is one use, not the definition. A smaller student can reduce deployment memory and inference work. A same-size student can consolidate specialists, recover instruction following or move useful behavior into another checkpoint. Self-distillation uses related checkpoints or the same model family; it need not reduce parameter count.

First state the desired transfer: task knowledge, response style, reasoning strategies, calibration, tool use or cleaner labels. Then test that property independently. A student can inherit confident wording without inheriting factual accuracy. Matching a teacher is not evidence that the teacher is correct.

Keep alternatives, not only the winning label

A hard classification label says which class wins. Its cross-entropy is −log p(y|x). Every non-target class receives the same target probability: zero. The label does not describe ambiguity or similarity among those alternatives.

Suppose a teacher assigns cat=.75, dog=.20 and car=.05. That target asks the student to distinguish two different errors: confusing cat with dog and confusing cat with car. These assigned numbers illustrate the signal; they do not prove that a real model understands semantic similarity.

Soft targets can provide useful uncertainty and class structure. They can also transfer a teacher's mistaken certainty. A teacher that puts almost all mass on the winning label gives little additional information compared with a one-hot target.

Align the teacher and student distributions

Let $z_T$ and $z_S$ be teacher and student logits over the same ordered label set. Logits are unnormalized scores, not probabilities. With temperature $\tau>0$, define $q_\tau=\operatorname{softmax}\left(\frac{z_T}{\tau}\right)$ for the fixed teacher and $p_\tau=\operatorname{softmax}\left(\frac{z_S}{\tau}\right)$ for the trainable student.

The teacher does not receive a gradient in ordinary distillation. Its checkpoint, vocabulary ordering and temperature define the target. Use log-softmax rather than computing tiny probabilities and taking their logarithms. For different vocabularies, index-by-index KL has no meaning without an explicit alignment method.

Cross-entropy and forward KL

Teacher entropy H(q) is constant with respect to the student when q is fixed. Forward-KL minimization and soft cross-entropy therefore have the same student gradient. Their reported scalar losses differ. Do not compare those loss values as though they were interchangeable.

Reverse KL is different: $D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i(\log p_i-\log q_i)$. The student supplies the weights. The student entropy is now trainable, so dropping it changes the gradient. See Losses and Objectives and Information Theory for the coding interpretation.

When matching the teacher still produces a gradient

A correct distillation objective can still have an inaccurate floating-point derivative. This matters most near a match, when the true update should be tiny. Barron et al. (2026) call this softmax cancellation; it is distinct from the high-confidence collapse described in Losses and Objectives.

Keep this note's convention: q is the fixed teacher and p is the student, both evaluated at the same temperature and vocabulary ordering. Before temperature weighting, the exact soft-target logit gradient is p−q. Computing the two probability vectors separately can lose their small difference. A forward softmax and a backward exp(log-softmax) can also follow different rounding paths, leaving a residual even at identical logits.

The cross-entropy value need not be zero here: it equals the teacher's entropy. The student gradient and forward KL should be zero in exact arithmetic. A nonzero scalar cross-entropy is not evidence that the matched student needs further training.

Preserve the difference before probabilities erase it

The paper takes teacher logits as an input to the loss. It starts from the student–teacher logit difference and evaluates the probability difference using expm1 and log1p. These functions preserve small deviations near zero better than forming exp(x)−1 or log(1+x) directly.

The construction then removes residual zero-sum drift with:

# Pedagogical gradient calculation; p = student, q = fixed teacher.
# z_s and z_t below are already temperature-scaled logits.
delta = z_s - z_t
d = delta - (q * delta).sum(-1, keepdim=True)
s = d - log1p((q * expm1(d)).sum(-1, keepdim=True))
u = q * expm1(s)
g = u - p * u.sum(-1, keepdim=True)

This computes a gradient, not a scalar loss that ordinary autodiff should differentiate again. The paper uses a stop-gradient construction to retain the forward cross-entropy while injecting g into the backward pass. A custom backward needs a deliberate contract: its reference implementation targets the student gradient with a fixed teacher, not arbitrary higher-order derivatives or a trainable teacher.

Here g differentiates with respect to the temperature-scaled student logits. With this note's $\tau^2$ objective weight, the gradient with respect to the original student logits is $\tau g$. Keep masks and reduction factors too; do not apply temperature scaling twice.

A sanity check before a long distillation run

1. Copy teacher weights into the student. Match tokenizer, input, temperature and support; disable dropout and other stochastic differences.
2. Detach teacher targets and inspect the student gradient on a nonempty valid mask before taking an optimizer step. Compare exact equality and small representable logit perturbations against a higher-precision reference.
3. For a no-drift optimizer test, also disable weight decay and other losses, and reset momentum/optimizer state. Otherwise parameters can move despite a zero current data gradient.
4. Repeat with the actual loss kernel, compiler and intermediate dtypes. FP32 reduces many errors but does not prove cancellation is absent.

In the paper's controlled test, the teacher solves modular addition modulo 53 and the student starts from the same weights. Adam runs for 2,000 steps with learning rate 0.002, ε=10⁻⁸, no weight decay and temperature 4. The tested standard loss produces student drift, strongest in BF16; the corrected loss preserves the matched solution. This is a diagnostic experiment, not evidence that every real distillation run suffers a large accuracy penalty.

Derivation and reference backward ↗ · Controlled matched-student experiment ↗. The usual log-softmax/FP32 sketch later in this note remains a useful baseline, not a guarantee of exact near-match gradients.

Expose preferences with temperature

Increasing a shared temperature softens both distributions. The student then receives more signal about lower-ranked alternatives. Decreasing temperature concentrates the target near its largest logit. Temperature changes the target and gradient; it is not merely a display setting.

Without $\tau^2$, the derivative is $\frac{p_\tau-q_\tau}{\tau}$. At high temperature, the probability difference itself often shrinks roughly as $\frac1\tau$ for bounded logit differences. The conventional τ² factor compensates that approximate scale change. It does not make the training trajectory exactly temperature-independent.

Choose temperature using held-out behavior and stable optimization, not only training loss. Softening an overconfident wrong teacher can change how its error transfers; it does not establish a correct target.

Keep labels and teacher targets separate

Hard labels anchor the known answer. Soft targets add the teacher's preferences among alternatives. When labels are incomplete, a stronger teacher may help. When the teacher is unreliable, a smaller teacher weight can reduce its influence. Neither observation supplies an automatic α rule.

Check label quality and teacher quality separately. Changing α also changes relative gradient scale. Compare matched target exposure, temperature, token masks and reduction before interpreting a gain as better knowledge transfer.

Repair labels without treating agreement as truth

A teacher can improve the training labels themselves. In information extraction, an ensemble may propose structured labels that the original annotation omitted. The ensemble label-repair recipe adds a candidate predicted by every ensemble member and flags an annotation predicted by none.

Treat those decisions as candidates for review, not proof. Ensemble members can share data, architecture and failure modes. Unanimous false predictions remain false. Removing a rare valid annotation because every teacher missed it can erase useful training support.

Use held-out annotated examples, confidence checks and selected human review. Preserve provenance for added, removed and downweighted labels. Teacher-generated SFT data, verifier-filtered solutions and synthetic preference pairs use a related idea: improve targets before fitting the student. The signal differs from direct logit matching.

Su Jianlin's information-extraction article ↗ supplies the original label-repair context.

Move from classes to next-token distributions

For an autoregressive language model, each prefix h supplies a classification problem over the vocabulary V. Teacher and student logits usually have shape [B,T,V]. A valid token loss compares q(v|h) and p(v|h) at the same input, token position and vocabulary index.

Shift logits and targets exactly once. For response-only training, a response-start token can be a target even when its preceding input is a prompt token. Excluding prompt targets is not the same as excluding prompt context.

Full-vocabulary targets are rich but expensive to store, transmit and normalize. Alternatives include sampled teacher text, selected positions, truncated distributions and sampled-action scores. A sampled teacher action estimates forward cross-entropy at a fixed prefix; changing who generates that prefix changes the training distribution.

Feature distillation instead aligns hidden representations. Different widths require a declared projection or another matching rule. Different layers need not represent the same operation. A projection can make dimensions agree without making the features semantically aligned.

What top-k targets omit

Keeping only a teacher's largest k probabilities reduces storage and traffic. But the retained entries may not sum to one. Renormalizing them conditions the target on belonging to that set. It no longer describes the teacher's full vocabulary distribution.

A remainder bucket can preserve excluded probability mass. It still omits the teacher's ranking inside the tail. Selected-token training additionally changes which positions receive supervision. Record both the vocabulary approximation and the token mask.

A sampled log-probability interface avoids transmitting the full distribution. It returns the teacher's score for the exact action the student took. That supports a sampled reverse-KL-style update, not an implicit full-vocabulary forward-KL target.

A stronger answer is not the same as a faithful trace

Teacher-generated reasoning traces are supervised examples. Their utility depends on answer quality, trace quality and the student's ability to fit them. A fluent explanation can accompany a wrong answer, so trace length is not a quality certificate.

Nanbeige4 separates final-solution refinement from reasoning reconstruction. It generates candidate answers, applies task-specific checks, critiques weaknesses and revises solutions. It then reconstructs a reasoning trace that leads to the refined answer.

That process can improve useful training examples. But a reconstructed explanation is not necessarily the actual causal reasoning used during generation. Check consistency with the final answer and external verification. Do not interpret the trace as direct evidence about hidden internal computation.

Who generates the prefix?

Trace SFT normally feeds the teacher's earlier tokens to the student. At deployment, the student feeds its own earlier tokens back into the model. An early student mistake can produce a prefix absent from the teacher's training examples. Later decisions then occur in unfamiliar states.

On-policy distillation generates histories with the student and queries the teacher at those histories. This reduces one source of train/deployment mismatch. It does not eliminate differences caused by changed templates, temperature, tools, truncation or deployment tasks.

Score exactly the history the student visited

The teacher must see the same prefix as the student: prompt, generated reasoning, prior turns, tool results and relevant special tokens. Scoring only the final answer is not equivalent to scoring that answer conditional on the full preceding trajectory.

A teacher can penalize the first token that leads toward a bad argument. It may assign high probability to a wrong final answer once the student has already supplied the wrong premise. Conditional predictability is not final-answer correctness.

Keep task verification alongside teacher feedback when correctness matters. A dense token score can guide local choices; a verifier can reject an entire incorrect solution. Do not label every large log-probability gap as the causal error that determined the outcome.

Dense teacher feedback and task outcomes

A terminal verifier supplies one outcome for a whole response. Policy-gradient credit spreads that outcome over the sampled decisions. Token-level teacher scores supply more local comparisons, but those comparisons optimize resemblance to a teacher rather than the task directly.

The often-used low-bandwidth description of sparse reward is a useful contrast, not a universal information theorem. Rewards can be continuous, structured or intermediate. The pretrained policy and repeated trajectories also matter.

Task reward and teacher targets can be combined. The objective then trades verified success against teacher resemblance. Tune that tradeoff on held-out tasks. A teacher may favor a familiar style that is unnecessary for success. See Reinforcement Learning for credit, likelihood ratios and policy-version requirements.

Two KL directions, two weighting measures

Forward KL strongly penalizes missing teacher mass. Reverse KL strongly penalizes student mass where the teacher assigns little probability. Their minima agree at p=q when the student can represent q. Their behavior differs when capacity, optimization or parameterization constrains the student.

Mode-covering and mode-seeking are shorthand for that constrained approximation. Reverse KL does not force an expressive student to choose one mode from a multimodal teacher. Likewise, a forward-KL fit does not inherently hallucinate: the compromise depends on the feasible student distributions.

Low KL measures agreement with the teacher, not external correctness. If the teacher reliably gives a wrong answer, copying that answer can achieve zero KL. This is the limit of calling the objective unhackable: the guarantee concerns its declared teacher distribution, not factual accuracy or safety. True zero teacher probability with positive student mass makes reverse KL infinite. Ordinary finite-logit softmax gives positive probabilities, but practical truncation can create hard zeros. Very rare sequences can also be effectively absent from a finite rollout budget. Distinguish exact support from practical exploration.

A sampled KL value is not a training loss

A sampled $d(a)$ estimates the KL value. Directly backpropagating through that sampled value does not estimate the KL gradient. With the sampled action held constant, $\nabla d(a)=\nabla\log p(a\mid h)$. Its expectation under the student is zero. The missing term comes from the parameter-dependent sampling probabilities.

There are two valid routes here. Sum over the complete vocabulary and differentiate the exact distribution. Or use sampled actions with the appropriate score-function weight. The implementation must state which route it uses.

An early token changes later states

In a sequence, a first-token update changes which later prefixes the student visits. The KL chain rule writes sequence KL as expected conditional KLs, but the expectation depends on the student's parameters.

Immediate-token feedback omits future visitation credit. It can still be a useful practical update. But it is not generally the full sequence-KL gradient. Variable lengths, EOS, normalization and truncated trajectories need additional explicit contracts.

GKD separates history collection from divergence

Generalized Knowledge Distillation (GKD) makes two choices explicit: which histories are collected and which distribution discrepancy is minimized. A mixture parameter λ selects between fixed examples and fresh student-generated sequences.

Consequently, on-policy collection does not imply reverse KL. A forward-KL vocabulary target can be evaluated at a student's prefix. The teacher/student sampling axis and divergence axis are independent.

GKD assumes a student that generates useful enough sequences for feedback. SFT can provide that initialization. The best divergence depends on the task and desired diversity. A generalized Jensen–Shannon mixture provides another tradeoff; it is not automatically superior.

GKD, §3 ↗ explains the stopped-history update.

Turn teacher scores into a declared update

For sampled-action feedback, record behavior log probability when collecting the token. Form a detached advantage $A=\log q(a\mid h)-\log p_b(a\mid h)$. Here $p_b$ is the behavior policy used for collection. At a matching current/behavior policy, a loss $-\mathbb E[A\log p_\theta(a\mid h)]$ has the correct fixed-prefix reverse-KL gradient.

An importance-sampling surrogate uses $\rho=\frac{p_\theta(a\mid h)}{p_b(a\mid h)}$ and maximizes $\rho A$. At $\rho=1$, its local derivative has the same direction. Clipping limits selected update branches; it changes the objective away from the exact unclipped estimator. Stale prefixes require more care than a single-token ratio.

Thinking Machines uses immediate per-token feedback, effectively discount zero. MiniLLM starts from sequence reverse KL and separates local feedback from future credit. Its practical method also uses teacher-mixed sampling, approximate weights and length normalization. These are related methods, not identical gradient contracts.

Detach the teacher scores and the declared advantage. Do not accidentally differentiate the student probability inside a reward that the optimizer assumes is fixed. Keep task rewards, KL regularization and teacher targets distinct in logs.

Count generation, scoring and communication

Teacher trace generation is autoregressive. Teacher scoring can process a known student sequence in one teacher-forced forward pass. That pass still attends over the sequence and evaluates required vocabulary scores. It is not constant-cost work.

A smaller student may collect trajectories cheaply. A same-size or inefficient student may not. Partial rollouts can expose local feedback before a complete answer exists, but they do not reproduce a full task-outcome signal.

Compare matched quality, prompts, token budgets and evaluation harnesses. Report theoretical FLOPs separately from GPU hours and latency. Parallel teacher scoring can change wall time without changing the FLOP estimate.

A historical reasoning comparison

Historical Thinking Machines setup: Qwen3-8B-Base, initialized by 400k-prompt SFT, reached approximately 60% AIME'24. OPD reached 70% in about 150 steps: 77k prompts, four samples each. SFT toward 2M prompts was extrapolated, not run.

The OPD teacher was Qwen3-8B; cost accounting used Qwen3-32B FLOPs. The reported 9–30× comparison depends on whether teacher data-generation costs are counted. These values are not universal efficiency multipliers.

Thinking Machines reasoning experiment ↗ reports the historical comparison. Updated reproduction models do not change these historical measurements.

Why targeted feedback can help adapters

At rank 32, the historical article reports a LoRA/full-fine-tuning gap of 13% after SFT and 6% after OPD. Keep the article's labels; the figure must establish any percentage-point interpretation.

One hypothesis is that feedback on the student's own errors requires less unrelated teacher-trajectory information. That could help a constrained adapter. The observation does not isolate that cause. Data size, batch size, initialization and optimization also differ.

Compare LoRA and full fine-tuning under matched exposure, learning-rate sweeps and evaluation. A reduced gap is not proof that adapter capacity is unlimited. See SFT and Adaptation for rank, scaling, trainable modules and deployment contracts.

Learn new facts; restore old behavior

An earlier assistant can preserve old behavior after domain midtraining supplies new facts. Teacher feedback on general prompts restores instruction following; it does not supply facts absent from that teacher.

This is a private-domain experiment, not a general retention guarantee.

For your own adaptation, track new-domain accuracy and old-task behavior independently. Collect general instruction prompts, student responses and teacher scores. Keep private contexts inside the permitted teacher boundary. A convenient external API does not authorize sending confidential data.

Repeated knowledge-acquisition and behavior-restoration stages are a plausible continual-learning workflow. Evaluate each cycle; an average chat score can hide lost capabilities or new factual errors.

Why self-sampled SFT can still move the policy

Suppose the initial student equals the teacher and we sample a finite dataset from that policy. The expected initial SFT score gradient is zero. The empirical token frequencies usually differ from the generating probabilities, so a particular batch can still update the model.

After that update, the fixed dataset was generated by an older policy. Repeated fitting is now off-policy relative to the updated student. Finite-sample drift is a possible explanation for degraded instruction following in the historical self-sample experiment. It is not a theorem that every self-sampled SFT run must degrade.

At exact teacher/student equality, an exact conditional-KL gradient is zero. Sampled log-ratio feedback is also zero pointwise when identical scoring contracts yield identical log probabilities. Numerical differences, replay, regularization and other losses can break that condition. Fresh rollouts alone do not guarantee convergence.

Search once, then transfer the resulting behavior

RL can discover useful behavior through expensive exploration and verification. Once a good teacher exists, distillation can transfer that behavior to another student. The student need not repeat every unsuccessful search trajectory.

A matched historical Qwen3-8B-Base experiment with rank-128 adapters reported 7–10× fewer steps and 50–100× compute efficiency for cloning an RL policy. Context and batch accounting matter.

Teacher training still costs compute. Sharing one teacher across many students can amortize that cost; it does not erase it. Test whether the student retains the discovered strategy on new problems, rather than only copying wording on familiar prompts.

Continue an RL climb through a supervised reset

A long RL climb can produce valuable responses while its optimization state becomes unstable or its behavior becomes too narrow. A supervised reset extracts good traces, trains a fresh or cleaner student checkpoint, and resumes RL.

The destination may be a newer pretrain/midtrain checkpoint or a native chat interface. Moving behavior through examples is different from averaging teacher weights. Templates, thinking channels, tool schemas and context policies must match the destination.

MAI describes this role as continuation between climbs. Rollback alone may fail when an earlier checkpoint already contains the problem. Useful responses can provide recoverable behavioral evidence, but rebuilding from those responses does not automatically fix a numerical, data or off-policy root cause.

MAI-Thinking-1 report ↗ describes the climb and reset context.

Select traces by quality and diversity

The MAI material reports that approximately O(1M) useful reasoning traces can preserve gains in its setting. Later strong checkpoints supplied better traces than very early climb states. Do not turn that observation into a universal minimum corpus size.

At a fixed token budget, prompt diversity was more useful than repeatedly sampling the same prompt. Successful traces alone can work when enough varied successes exist. They can also discard recovery behavior and hard-task coverage, so inspect the filtered distribution.

Using only the final checkpoint can be weaker than sampling several strong late checkpoints. Broader behavior is a plausible explanation, not a controlled proof. Select checkpoints by verified quality and complementary coverage; chronology is not a quality metric by itself.

Retain source checkpoint, prompt family, outcome, generation settings and filtering decision for each trace. Deduplicate without removing genuinely different strategies. Hold out prompt families, not only text rows.

Do not forget the long-context distribution

A short-trace self-distillation dataset can underrepresent long-context behavior. The student may retain short-task gains while losing retrieval, long reasoning or tool state across long histories.

Mix suitable midtraining or long-context examples into the reset. Evaluate length buckets and long-history tasks before restarting RL. A token budget is not the same as a distribution over sequence lengths: many short examples cannot test long-context behavior.

Long examples cost more and may be harder to batch. Set the mixture using retained capability, not a fixed universal fraction. Context extension and positional changes remain separate architectural/training contracts.

Consolidate domain and effort specialists

Kimi K3 first trains specialized teachers, then consolidates nine teachers into one student. Domains include reasoning/vision, search/knowledge work, general agents and coding agents. Effort levels provide another conditioning axis; nine does not imply every domain/effort combination is present.

The student generates the trajectory. The selected teacher scores those exact tokens. Specialists can improve independently before consolidation. Shared prompts may receive conflicting preferences across teachers; domain routing does not prove the absence of interference.

Sampled-token scores and top-k distributions provide different signals. The K3 consolidation description alone does not establish which target format is better under a matched budget. Do not infer a universal advantage from the interface used by one report.

Frontier Recipes retains the complete K3 training context; K3 report ↗ is the primary reference.

Consolidate heterogeneous teachers with full targets

DeepSeek-V4.1-Flash's final stage consolidates over 40 teachers using full-vocabulary on-policy distillation. Teachers can differ in architecture and development stage. Dataset mixture, active teachers and per-dataset concurrency can change as capability evaluations evolve.

The student supplies the histories; teacher distributions supply targets at those histories. That is not fixed-trace SFT, nor the intermediate parameter merges used for RL initialization. Full-vocabulary supervision gives alternatives beyond the single sampled token.

The report does not disclose every loss coefficient or teacher-to-domain assignment. Heterogeneous architectures are compatible with output-level transfer, but tokenization and output semantics still need an alignment contract. Do not invent undisclosed settings.

DeepSeek report, §5.2.4 ↗ describes consolidation; see Frontier Recipes for the surrounding recipe.

Combine preference pairs with local teacher targets

Nanbeige4's Direct Preference Distillation (DPD) uses filtered Nanbeige3.5-Pro positives and responses from the 3B student as negatives. It retains pairs only when the negative is clearly worse. The negative branch therefore reflects relevant student failures.

Local teacher targets can reveal confidently wrong tokens and underestimated alternatives. The preference margin alone provides response-level ordering. Combining the signals can teach both the chosen solution and the distribution around failure states.

Relative to SFT, the original report summary records roughly 8% AIME, 10% GPQA, 30% BFCL V4 and 8% Arena-style gains. These labels do not establish percentage-point units. The exact tables must determine absolute scores and relative changes.

RL initialized from the distilled checkpoint performed better than RL initialized directly from SFT. The available DPD section does not isolate each joint-loss component as cleanly as the schedule ablation. Treat attribution accordingly. See Compact Model Recipes for the architecture and broader training recipe. Nanbeige4-3B report ↗ is the primary reference.

A practical collection and evaluation contract

Start with the destination task and deployment contract. Check that the teacher is stronger on held-out cases that matter. A benchmark average can conceal a weak teacher in a critical subdomain.

If the student rarely produces relevant trajectories, supply useful SFT examples or domain midtraining first. Softmax positivity alone does not guarantee practical coverage of a long useful sequence. Then collect fresh student states and use the declared teacher signal.

Evaluate task success, retention, calibration, diversity, response length and budget. Include student failures, recovery paths, long contexts and tool interactions. Compare intermediate checkpoints and independent prompt families. Monitor teacher disagreement, KL tails and acceptance/filter rates alongside mean training loss.

Two small kernels with different gradients

This first pseudocode kernel uses aligned teacher/student logits [B,T,V]. It computes a forward-KL-style soft cross-entropy; teacher entropy is omitted because the teacher is fixed. The mask excludes declared non-target positions. It is a baseline, not a guarantee of accurate gradients near an exact teacher–student match; see the softmax cancellation section.

with no_grad():
    q = softmax(teacher_logits.float() / tau, dim=-1)
logp = log_softmax(student_logits.float() / tau, dim=-1)
per_token = -(q * logp).sum(-1) * tau**2
loss = (per_token * mask).sum() / mask.sum()
loss.backward()

Require a nonempty mask rather than silently inventing a denominator. This allocates the full vocabulary target; use chunking or another explicit approximation when memory requires it. Teacher-forced scoring does not mean the prefix came from the teacher.

This second conceptual kernel uses sampled student actions and detached local feedback. Behavior probabilities come from collection; current probabilities are recomputed at the same histories. Shapes are [B,T].

with no_grad():
    advantage = teacher_logp - behavior_logp
ratio = (current_logp - behavior_logp).exp()
per_token = -ratio * advantage
loss = (per_token * mask).sum() / mask.sum()
loss.backward()

At matching current/behavior probabilities, this estimates the fixed-prefix reverse-KL gradient. It is not an exact full-sequence update under arbitrary replay. Adding clipping, length normalization, task reward or future credit needs corresponding tests. Neither snippet is a production training loop.

Diagnose what transfer failed

Also check data leakage, narrow synthetic style and teacher hallucinations. Distillation can conceal inherited evaluation contamination because the student never directly saw the original contaminated corpus. Keep teacher provenance and evaluate genuinely new examples.

A weak or incompatible teacher cannot be repaired merely by increasing its weight. More synthetic data is not automatically more useful coverage. Prefer carefully verified targets and matched-budget experiments.

How the methods connect

These branches coexist. Full-logit targets, generated demonstrations and sampled-action feedback solve different access and cost problems. A newer consolidation report does not make earlier soft-target training obsolete.

Sources and related topics

Hinton et al.: Distilling the Knowledge in a Neural Network ↗ · DAgger ↗ · GKD ↗ · MiniLLM ↗ · Thinking Machines OPD ↗

SFT and Adaptation · Reinforcement Learning · Preference Optimization · Losses and Objectives · Frontier Recipes · Compact Recipes

The interactive models are exact constructed examples. They are not measured training curves or evidence that a particular loss will improve a real deployment.