Preference Optimization

Compare behavior instead of copying it

Supervised fine-tuning imitates selected demonstrations. Preference optimization uses a different signal: for the same prompt, response A is better than response B. This can express style, helpfulness or reasoning quality without specifying one exact answer.

Begin with a useful assistant checkpoint and test whether comparative feedback improves it. Direct Preference Optimization (DPO) is a convenient baseline. It avoids an explicit learned reward model and an online rollout loop during its standard offline training.

This makes preference training a useful middle step between imitation and online RL. It is not always cheaper or more stable: long paired responses, reference scoring, poor labels and aggressive updates can still make a run expensive or unreliable. A working SFT interface remains a practical starting point, not an absolute requirement for every objective. SFT and Adaptation →

Build a meaningful comparison

A preference pair contains a prompt x, chosen response y+ and rejected response y−. Both responses must answer the same conditioned task. Record the labeling criterion: correctness, instruction compliance, brevity, safety or a deliberate combination.

If a chosen answer is correct but verbose and a rejected answer is concise but wrong, the label does not isolate verbosity. Conflicting criteria can teach inconsistent behavior. Preserve ties or ambiguous cases under an explicit policy instead of forcing arbitrary winners.

Keep evaluator provenance, model versions and generation budgets. Deduplicate prompts and response families before train/evaluation splitting. Near-identical responses can supply weak differences; easy obviously bad negatives can fail to teach the distinctions the deployed model needs.

Small preference sets can already be useful. Their value depends on coverage and label quality, not a universal minimum count. Start with a representative pilot rather than waiting for a huge labeling pipeline.

Score the same response contract

Score each completion conditioned on the same prompt. Sum causal token log probabilities over the response mask, including the intended termination token. Exclude padding and prompt targets. Match templates, tokenizer IDs and one-position shifting between policy and reference.

A response can span thought channels and tool actions. Define which spans belong to its score before training. Do not include different histories in chosen and rejected prefixes while treating them as a simple comparison for one prompt.

Truncation can cut an answer or remove its ending. Check actual retained IDs and masks, not raw text lengths. The SFT reader → covers the shared serialization, masking and parser contract.

Measure improvement relative to a reference

The reference is frozen. At $\pi_\theta=\pi_{\mathrm{ref}}$, $\Delta=0$, preference probability is $\frac12$ and pair loss is $\log 2$. The two sequences need not have equal absolute probability at this point.

A lower loss means a better fitted comparison under this objective. It is not a direct measure of correctness, calibration or the probability of generating a useful answer.

A better ratio can hide a worse answer

Take three exhaustive toy responses: chosen, rejected and other. The reference assigns [0.5,0.3,0.2], while the initial policy assigns [0.4,0.4,0.2]. The initial margin is $-\log\frac53$, because the policy favors chosen less strongly than the reference.

On the second path at $t=4$, both named answers have lower probability than 0.4. Rejected fell faster, so Δ still rises. This demonstrates why ratio improvement alone does not prove that chosen answers became more likely. The example is a possible normalized distribution change, not a prediction of every DPO run.

Read the gradient without overclaiming

A misordered comparison has greater pressure than an already satisfied comparison. Both sequence scores share model parameters. Normalization, competing examples and parameter geometry determine the actual update.

Monitor absolute chosen/rejected log probabilities as well as the margin. The mathematical partial derivatives do not guarantee that one optimizer step increases every preferred sequence's probability. A useful preference model must improve deployed behavior beyond the training pairs.

Why the reference appears

The reference anchors the comparison to an existing policy. The conventional regularized objective maximizes $\mathbb E[r]-\beta D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{ref}})$. For fixed reward $r$, the optimum is proportional to $\pi_{\mathrm{ref}}\exp\left(\frac r\beta\right)$.

Substituting this relationship into a Bradley–Terry comparison yields the DPO pair loss. This connection requires its preference-model and support assumptions. Finite data and finite optimization do not guarantee recovery of a perfect online-RL solution. DPO derivation ↗

Keep the reference truly frozen, including adapters and trainable vocabulary rows. Disabling an adapter recovers the intended reference only if no shared base state changed. Cached reference scores are valid only for the same reference, tokens and score masks.

Two different questions about beta

Two statements about β answer different questions. With fixed reward in $\mathbb E[r]-\beta D_{\mathrm{KL}}(\pi\Vert\pi_{\mathrm{ref}})$, larger β penalizes departure more strongly. In the DPO classifier, β also scales the margin and its gradient.

At $\Delta=0$, the chosen score derivative is $-\frac{\beta}{2}$. Increasing β therefore increases that local gradient magnitude, even though larger β means stronger anchoring in the fixed-reward optimum. There is no contradiction: one describes a local training step, the other a solved objective.

Read the actual loss before interpreting configuration values. Some methods use inverse-temperature or margin conventions. The original playbook-based note suggests β in [0.01,0.5] as a search region, not a universal default or a guarantee of monotonic measured KL.

Compute a stable response-level loss

Compute token log probabilities with log-softmax, select response tokens and sum them per sequence. Score chosen and rejected with both policy and frozen reference. Then form the relative margin.

with no_grad():
    ref_plus, ref_minus = response_logps(reference, batch)
pi_plus, pi_minus = response_logps(policy, batch)
margin = (pi_plus - pi_minus) - (ref_plus - ref_minus)
loss_per_pair = -logsigmoid(beta * margin)
loss = loss_per_pair.mean()
loss.backward()

This is conceptual code. The helper must apply one shift and the exact completion masks. Use stable log-sigmoid rather than taking log(sigmoid(z)) after saturation. Accumulate long sequence sums with adequate precision.

State pair weights and reduction denominators. Do not silently replace sequence sums with token means when moving to another trainer. Check dropout policy and reference determinism before caching scores.

Choose the feedback before the acronym

After a working DPO baseline, change the objective only to answer a concrete feedback or optimization question. Alternative acronyms do not establish superiority.

APO separates cases where winners deserve promotion from cases where even winners should lose probability, with losers reduced more. The TRL zero/down implementations encode different assumptions. APO configuration ↗

ORPO combines supervised learning on chosen responses with an odds-ratio preference term, without a separate reference policy. It is not DPO with reference subtraction deleted. Its score normalization and penalty weight form a separate contract. ORPO ↗

KTO accepts individual desirable/undesirable labels rather than requiring paired rankings. Its prospect-theoretic objective uses a reference-relative baseline and separate label weighting. Use it when that feedback is available, not because every binary label is equally reliable. KTO ↗

A small controlled preference experiment

First verify pair direction, masks and reference identity on a few examples. At initialization equal to the reference, confirm zero margins. Reversing chosen/rejected should reverse the margin. Identical responses should supply no directional distinction.

The playbook-based original suggests learning rates roughly 5–20 times lower than SFT. Treat this as a sweep starting point. Effective batch, adapter/full tuning, response lengths, objective and optimizer affect the useful scale.

Keep a matched pilot: same base, examples, target preprocessing, update budget and held-out generation settings. Sweep learning rate and β jointly. A comparison that tunes one objective thoroughly but gives another a default is not a clean objective ablation.

Record the intended data coverage and evaluation budget. A modest set can reveal a useful gain; it cannot establish broad safety or reasoning reliability by size alone.

Stop on behavior, not pair loss

Preference fitting can overfit quickly. The original practical note warns that this can happen around one epoch. That is a warning to evaluate early, not a universal stopping threshold.

Track held-out pair accuracy and generated behavior separately. A larger implicit reward margin can coexist with worse response quality or lower preferred-response probability. The learned reward is reference-relative, not a calibrated external evaluator.

Short runs over shuffled slices can reduce repeated exposure during iteration. They do not remove leakage, bad labels or selection bias. Keep held-out prompts out of labeling and training decisions. Select checkpoints by downstream behavior under fixed evaluation conditions, not the lowest training loss.

Reasoning preferences need reliable outcomes

Preference training is not limited to tone or refusals. Correctness-based pairs can improve reasoning behavior and may make immediate escalation to online RL unnecessary. This is a practical opportunity, not a claim that offline pairs discover every missing strategy.

Check the criterion that selects the winner. Final-answer verification can miss wrong intermediate reasoning. A stylistic judge can favor long confident traces. Process labels and reliable final verifiers express different preferences.

Compare accuracy, response length, termination and general capabilities. Preserve failed actions as context when they explain recovery, but do not call a malformed successful trajectory a clean demonstration.

Offline, refreshed preferences and RL

Offline preference optimization fits a fixed set. Iterative or on-policy variants refresh candidate answers and labels as the model changes. New comparisons can target current failures and reduce mismatch with an old candidate distribution.

Refreshing data costs generation, evaluation and provenance tracking. It does not make preference fitting identical to online policy-gradient RL. The update objective and the data-collection policy are separate choices.

Use online RL when sampled actions and reward feedback provide something static comparisons do not capture, especially environment interaction. It brings reward design, credit assignment, rollout versioning and throughput constraints. This is not a mandatory final stage for every assistant.

Checks before trusting a preference gain

1. Define the preference criterion and keep ambiguous cases explicit.
2. Verify identical prompts, templates, causal shifts and completion masks.
3. Freeze the intended reference and version cached scores.
4. Check margins, finite loss and gradient direction on tiny examples.
5. Run matched learning-rate/β sweeps; monitor absolute probabilities too.
6. Evaluate held-out generations early, including reasoning, length and stopping.
7. Refresh data or change objectives only for an observed limitation.

Losses → owns logistic scoring and reductions. Information Theory → explains KL direction and coding cost. Training Data → covers mixture and leakage.

Sources and original context

DPO paper ↗ · Current TRL DPO documentation ↗ · ORPO ↗ · KTO ↗ · Smol Training Playbook ↗

The small distributions are exact illustrations. Learning-rate, β and epoch suggestions are source-attributed heuristics, not reconstructed experiments or universal guarantees.

Reinforcement Learning → · Distillation → · Reasoning Models →