Supervised Fine-Tuning and Adaptation

Decide which behavior to imitate

Supervised fine-tuning (SFT) trains a model to predict selected tokens from demonstrations. For an assistant, those tokens usually belong to assistant responses. History supplies the condition; the response supplies the behavior to imitate.

SFT can establish instruction following, reasoning formats, tool calls and turn termination. It also changes the prior over style and answers. A stronger pretrained checkpoint does not automatically know your assistant interface. An existing instruct checkpoint may need less adaptation, but its previous behaviors can conflict with yours.

Start with SFT when you need reliable demonstrated behavior. This is a practical default, not a theorem that every pipeline needs another SFT stage. Preference optimization compares responses. Online reinforcement learning evaluates sampled actions. Neither is the same objective as reproducing a chosen demonstration.

Test the smallest complete assistant

Choose a base with the intended size, dense/MoE layout, post-training track record and training/serving budget. A convenient proxy may hide deployment constraints. Before a large run, build a baby baseline: representative data, a short run and the actual deployment interface. Test system instructions, tools, reasoning modes, multiple turns and stopping. A training-loss curve alone cannot detect a wrong template or a missing end target.

Talk to the checkpoint directly. Ask a greeting, an equation, a tool request and a follow-up. If a greeting produces pages of derivation, demonstrations may reward inappropriate reasoning. If an equation stops at an empty thought tag, inspect targets before changing the optimizer.

Keep a held-out suite and fixed conversational traces. Compare base and adapted checkpoints under the same template and decoding settings. Isolate behavior changes rather than selecting a best-looking conversation after training.

Select demonstrations for the behavior

Choose examples that express the desired response policy. Dataset prestige and teacher size do not establish this match. Synthetic traces can contain wrong answers, unnecessary reflection and malformed tools. Successful output does not prove that every intermediate action is worth imitating.

Count supervised target tokens as well as examples. Long explanations can dominate token-mean loss despite a small example share. Equal source sampling does not imply equal target contribution. Define the mixture and reduction together. Training Data →

Separate new knowledge, task behavior and interface adaptation when interpreting gains. Keep unrelated capabilities in evaluation. A narrow dataset can improve its target task while reducing general performance. Base checkpoint choice and data diversity can matter as much as rank.

Turn messages into a trained interface

The training object is a token sequence, not role-labeled dictionaries. The chat template inserts boundaries and control markers. Tokenization maps the rendered sequence to IDs. Masks then select which targets receive loss.

Use the checkpoint's contract. ChatML and Qwen-style formats are design examples, not universal replacements for trained templates. BOS is a checkpoint-specific ID; writing “[BOS]” need not insert it.

Direct template tokenization avoids a second automatic BOS/EOS insertion. If you render text first and tokenize later, disable additional special tokens when the template already supplies them. For completed SFT conversations, do not append an unnecessary empty assistant generation prefix. Starting a new message differs from continuing the last message.

Inspect exact IDs, spaces, newlines and endings. Roles organize conversation; they do not guarantee security against prompt injection.

Choose targets, not just visible messages

Assistant-only training can imitate both responses in a two-turn conversation. Final-response-only training retains earlier turns as context but selects only the last response. All-role training also predicts user and system tokens. These optimize different behaviors.

A target at position t is predicted by logits at position t−1. The target mask is not the attention mask. Ignoring a user token's direct loss does not remove its context. Later losses can still backpropagate through its representation.

The default targets are 7, 8, 13, 14 and 15. Their predictors are 6, 7, 12, 13 and 14. The assistant header conditions the first body target. Whether headers and endings receive loss is an explicit choice. End targets teach termination.

Position zero has no preceding predictor here. In a real sequence, BOS can predict later tokens. Padding never contributes to this example.

Reduce over the right denominator

A token mean gives selected tokens equal weight. A mean of per-example means gives short responses more weight per token. A sum changes scale with target count. Across microbatches/devices, aggregate the intended numerator and denominator. Averaging local means can overweight small batches.

logits = model(input_ids).logits[:, :-1]  # [B,T-1,V]
targets = input_ids[:, 1:]                # [B,T-1]
valid = target_mask[:, 1:] & nonpad[:, 1:]
labels = targets.masked_fill(~valid, -100)
total = cross_entropy(logits.reshape(-1, V),
                      labels.reshape(-1), reduction="sum")
count = valid.sum()
loss = total / count  # handle count == 0 first

This conceptual snippet shifts once. Some model APIs already shift labels internally; do not shift twice. Ignore index −100 belongs in labels, not input IDs. If PAD and EOS share an ID, mask padding positions rather than removing every occurrence of that ID. Otherwise real termination targets disappear.

Packing changes context unless isolated

Packing fills one sequence with several examples. An end marker separates text but is not an attention barrier. With ordinary causal attention, later examples can read earlier ones. This is not equivalent to independent examples.

Independent packing needs sequence-aware block-causal attention and compatible positions. Targets also need boundary handling: do not predict a new example's first token from an unrelated previous example when independence is intended. Confirm the implementation, not just a packing flag.

Truncation can remove closing thought markers, tool results or endings, rewarding incomplete structures. Assistant-only library options also depend on templates exposing assistant generation regions. Verify masks on actual rendered samples; a role-aware flag is not proof of correct labels.

Packing also changes the optimizer clock

After checking attention boundaries, check the optimization budget. A batch of eight packed sequences can contain many more demonstrations than eight padded sequences. Equal sequence batches do not establish equal learning conditions.

Both arms see every demonstration once. Packing uses one quarter of the slots but also one quarter of the updates. Keeping 128 packed updates would instead repeat the data four times. Neither comparison isolates a kernel improvement.

To match this example's token batch and update count, use two packed sequences per update, not eight. This arithmetic assumes identical examples, no truncation, and isolated attention. It predicts neither wall-clock time nor quality.

Also count supervised targets. A packed input can contain long user messages with no direct loss. Preserve the target mask and loss reduction, then retune learning rate under a declared budget. Compare quality at fixed exposure and operational value at fixed accelerator time; report both.

The SmolLM3 SFT experiments report 3–5× throughput and up to 33× more non-padding tokens per update. Large-batch quality losses were specific to their model and dataset, not evidence that packing intrinsically harms learning.

Smol Training Playbook: To Pack or Not to Pack? ↗ · Choose the comparison budget →

Preserve reasoning and tool conventions

Decide whether previous reasoning traces stay in later-turn history. Keeping and stripping traces create different conditional inputs. Apply the declared policy during preparation, rollouts and deployment. Hiding reasoning in the UI does not require stripping it from model history.

Tool names, schemas, call IDs and results are part of the contract. Preserve observations that justify later actions. Keep a failed action as recovery context while masking its direct imitation target when that action is undesirable.

Checkpoint-specific markers matter. Gemma 4’s reported markers distinguish checkpoint type, mode and channel.

Format correctness differs from answer correctness. A correct answer in an invalid channel can fail a harness or remain invisible. Compact Model Recipes →

Parse independently of chunk boundaries

A streamed delta need not equal one token. Runtimes can buffer several tokens or split decoded text. A delimiter can cross deltas; several delimiters can arrive together. Parsing each delta independently misses these cases.

Maintain pending text and parser state. Consume complete boundaries, emit the correct channel, and retain the unmatched suffix. Test a complete stream under many chunk partitions. Parsed messages should not change when only chunk boundaries change.

Include quoted delimiters, repeated markers, escaped text and truncated endings. Distinguish control-token IDs from ordinary text resembling their spelling. Display filtering is not a secure tool boundary. Parser and tokenizer must agree about control.

Adding a token changes two interfaces

Reasoning tags, tool-call tags and XML-like structure markers can require new IDs. A new marker can represent one atomic control ID instead of several old text pieces. Adding its spelling to a prompt does not register that ID. Register it, resize input embeddings and handle the output head.

Record old subtoken segmentation before registration. Afterwards the same string may resolve to the new ID, defeating subtoken initialization. Save tokenizer, template and resized state together.

Untied input embeddings and output heads are separate matrices. Both need explicit initialization and training policies. A poorly initialized row can introduce unusual activations or logits. In MoE, these can change routing; dead experts are a possible failure, not an inevitable result.

Initialize new rows without guarantees

Subtoken averaging starts near known representations. It does not prove the desired semantics. Opposing vectors can cancel and produce small norms. Inspect norms and initial output probabilities, not only whether the mean exists.

Even with unchanged old logits, adding an output row changes normalization. Let $Z=\sum_{i\in\mathcal V_{\mathrm{old}}}\exp(z_i)$ be the old partition and $a=\exp(z_{\mathrm{new}})$ the extra unnormalized mass. Every old probability is multiplied by $\frac{Z}{Z+a}$.

Extending the old distribution with zero probability on the new ID gives $D_{\mathrm{KL}}(p_{\mathrm{old}}\Vert p_{\mathrm{expanded}})=\log\left(1+\frac aZ\right)$. This measures fixed-logit normalization drift. It does not establish full-model stability after changing an input row.

Adapt vocabulary before the main run

The Laguna report describes short vocabulary adaptation: freeze the main body, train input/output embeddings, then perform main adaptation. This aims to stabilize the new interface before the body also changes. Exact warmup hyperparameters are not reconstructed here.

Training all embedding rows differs from training selected new rows. With tied weights, avoid duplicate optimizer ownership. With adapters, include vocabulary rows and any trained head in saved state; adapter matrices alone may not contain them.

Check formatting, termination, gradient norms and old tasks before unfreezing. Stable loss is insufficient when new markers are never generated. Explicit vocabulary warmup differs from LoRA's implicit early branch schedule.

Choose full fine-tuning or adapters

Full fine-tuning (FullFT) updates base weights. Low-rank adaptation (LoRA) freezes them and trains factor matrices. Required capacity, data scale, memory, module coverage and serving layout determine the choice.

Do not compare attention-only adapters with full-model training and attribute every gap to rank. Match target exposure, tuning and checkpoint selection. Different batch sizes or precision introduce additional differences.

Follow the low-rank branch

A maps input to r coordinates; B maps them to output width. The update has rank at most r. Its $r(d_{\rm in}+d_{\rm out})$ trainable entries replace $d_{\rm in}d_{\rm out}$ base entries. Savings require small rank relative to both widths.

With dropout disabled and compatible arithmetic, $W_{\mathrm{merged}}=W+\gamma BA$ gives the same linear result. Runtime adapters avoid a full checkpoint per task; merging removes the separate factor branch. Quantization and rounding can break bitwise equality.

Why zero B still learns

Standard initialization uses nonzero A and zero B. The adapter initially outputs zero, retaining the base computation. Let g be the loss gradient with respect to y.

At B=0, A gets no adapter-path gradient initially. B can learn because Ax is nonzero. After B changes, A receives that path's gradient. Decay or other objectives can still move A initially.

Freezing W removes its parameter gradient, not every backward operation. Earlier trainable layers still require gradients through W. Small optimizer state therefore does not imply negligible backward work.

Capacity is not only the adapter rank

LoRA Without Regret sweeps rank, learning rate, layer coverage, data and batch across Llama/Qwen experiments. High rank can approach FullFT. Low-rank curves may follow FullFT early but require more data later. This is not automatically a permanent loss floor.

Attention-only adaptation can miss useful feed-forward updates. MoE expert adaptation adds conditional capacity, but only activated experts receive an example's gradients. Compare matched trainable counts before attributing differences to rank alone.

The heuristic P≳I describes enough adjustable capacity for task information. Parameters and information bits are not interchangeable units. This is not a formula for choosing rank from dataset bytes.

Parameter placement changes available update geometry. This is an explanatory hypothesis, not proof of identical nonlinear trajectories or complete causation for every measured gap.

Separate rank from update scale

BA sums r rank-one outer products. Standard $\frac\alpha r$ compensates that sum under particular initialization and early-update assumptions. Expected early behavior can be approximately rank-independent; whole trajectories need not match.

Changing rank without documenting α, initialization and learning rates confounds capacity with scale. Rank-stabilized LoRA instead uses $\frac\alpha{\sqrt r}$: a distinct scaling specification.

Record adapter dropout, target modules and separate A/B learning rates. Verify the implementation rather than assuming every configuration called LoRA makes the same choices.

Compare equivalent parameter coordinates

Under an idealized Adam argument, multiply init_A and LR_A by p and LR_B by q. With compatible optimizer states and negligible epsilon effects, this symmetry removes redundant tuning directions.

Choose positive rescaling factors $p=2$ and $q=3$. The following assigned settings preserve both effective tuning coordinates.

Both settings give $c_{\mathrm{initial}}=0.0002$ and $c_{\mathrm{time}}=100$. This is coordinate arithmetic under the stated optimizer assumptions, not a measured training comparison.

Epsilon, clipping, decay, precision and unmatched dropout/state can break ideal equivalence. Use the symmetry to reason about sweeps, not to assume all library runs are invariant.

Tune optimization, not only rank

The report's practical experiments often favor a LoRA learning rate around ten times tuned FullFT. A fourteen-case Llama/Qwen fit was close to 9.8×. The roughly 15× suggestion for very short runs is preliminary, not a universal constant.

Starting from B=0 makes the effective branch grow initially. This produces an implicit schedule even with constant optimizer learning rate. It is not identical to explicit warmup.

Larger rank does not always remove large-batch gaps. The authors discuss beneficial gradient noise as an optimization explanation, not a universal rule. Sweep batch and learning rate separately with matched target exposure.

Some displayed curves are pointwise minima over learning-rate sweeps. Different training points can select different runs. Do not read that envelope as one realizable schedule. Optimization →

Why some RL updates need less rank

SFT supplies many chosen token targets. Terminal-reward policy gradients instead combine a score gradient S with scalar advantage Adv. The report finds rank-one parity in some math-RL experiments and proposes feedback bandwidth as an explanation.

The independence condition matters. Shared rollout statistics or a different conditioning choice can alter the argument. A continuous scalar is not literally one bit. The finite-bin entropy bound does not convert parameter counts into information bits.

Zero advantage supplies no update through this channel. Process rewards, auxiliary losses and model-based feedback change the signal. Rank-one parity remains task/algorithm-specific, not a guarantee for every RL method or agent environment.

Small optimizer state is not free backward

For one square N×N matrix, count multiply–adds and ignore other work. FullFT needs forward, input-gradient and weight-gradient products: about 3N². LoRA retains two base products and adds factor work: about 2N²+6Nr.

At N=4096 and r=16, the ratio is 0.6745. Trainable entries fall from 16,777,216 to 131,072. The limiting work ratio is two thirds, not zero. Base weights, activations, attention and communication remain.

Kernels, batching, checkpointing and hardware determine latency. Quantized-base adaptation reduces weight storage but needs an explicit backward/merge precision contract. A small adapter file does not imply a similarly small runtime footprint.

Export the entire trained contract

Save base revision, tokenizer, template, target modules, rank, α/scaling convention, dtype and every trained nonadapter tensor. Include new vocabulary rows and heads. Record weight tying and expert ownership.

Reload in a fresh process. Compare representative logits, generated IDs, stopping and tool/reasoning channels. Compare merged and unmerged paths with dropout disabled and declared tolerances. Successful tensor loading does not prove interface compatibility.

Rebuild the same multi-turn prefixes as training/rollout tests. Test parser partitions. For multimodal models, save processors and input markers. Identical weights can behave differently behind a changed interface.

With multiple adapters, reject incompatible vocabulary/template assumptions. A shared base does not make adaptations mutually interchangeable.

A reliable adaptation sequence

1. Specify behavior and select a compatible base.
2. Render histories; inspect IDs, boundaries and modes.
3. Verify one shift, targets, padding and reduction.
4. Verify packing, truncation and trainable/saved state.
5. Run a complete small baseline and held-out tests.
6. Sweep rank, module coverage, learning rate and batch with matched target exposure; evaluate unrelated capabilities.
7. Reload export and retest prefixes, channels, stopping and numerical agreement.

This order reveals interface failures before expensive capacity experiments. If output never stops, inspect end targets and serving stops first. Higher rank cannot repair a target that never receives loss.

Losses → explains reduction; Training Stability → separates recoverable training from useful generalization.

Sources and connected topics

LoRA Without Regret ↗ · Original LoRA paper ↗ · Gemma 4 interface and Table 11 ↗ · Transformers chat templates ↗ · TRL SFT trainer ↗ · PEFT configuration ↗ · Laguna report ↗

Laguna’s warmup is a report-specific recipe; undisclosed settings are not inferred. The probabilities and matrices are analytical examples, not measured training outcomes.