Reinforcement Learning

From a response to a decision process

A demonstration specifies actions to imitate. Reinforcement learning (RL) instead samples actions and evaluates their consequences. The model must change the probability of useful behavior without a supplied answer for every decision.

Pretraining fits observed text. Supervised fine-tuning (SFT) supplies demonstrations and an interface. RL evaluates sampled choices against outcomes. These stages have different objectives; not every recipe requires all three. Reasoning, tool use, stopping, instruction following and safety can all be reward targets, but a task reward does not measure every capability.

In a Markov decision process, state s contains the information needed for the next decision. Policy π(a|s) chooses action a. The environment returns reward and a new state. Return is the discounted sum of future rewards.

For an autoregressive language model, the state is the prompt and generated prefix. A next token is an action. Tool calls also change external state. A reward may arrive only after the final answer or many environment turns. Response length is then both a compute cost and a credit-assignment horizon.

Propagate future value

The Bellman equation expresses a long return through one transition and the value of the next state. Policy evaluation averages actions under π. Optimal control chooses the action with greatest expected return.

For finite states and bounded rewards, γ<1 makes the Bellman optimality operator a contraction. Repeated sweeps converge to its unique fixed point. Undiscounted γ=1 needs separate episodic or proper-policy assumptions; the contraction argument does not apply unchanged.

Value iteration needs a transition model. Each sweep evaluates the available actions using the preceding state values, then takes their maximum. Policy iteration instead evaluates a fixed policy and alternates that evaluation with greedy policy improvement. Q-learning uses sampled transitions.

Its off-policy target differs from following the sampled next action. A deep Q network approximates this table; approximation, replay and target networks do not inherit every tabular convergence guarantee.

Increase the probability of useful actions

For an LLM, directly optimizing its action probabilities is more practical than constructing a table over all prefixes. Differentiate the expected reward with the log-derivative identity.

A positive advantage increases sampled-action log probability; a negative advantage decreases it. The shared parameters also change other probabilities. This is not a guarantee that every successful complete response becomes more likely after one finite optimizer step.

A state-dependent baseline can estimate expected future reward. It must not depend on the sampled action in a way that changes the expectation without correction. The formula above describes the reward objective; adding entropy or KL creates additional terms.

Separate reward, credit and policy update

Before choosing an acronym, answer three independent questions. What is good? What was better than expected? How much may the policy change?

RL from human feedback (RLHF) describes a feedback pipeline. RL with verifiable rewards (RLVR) describes a reward source. Either can use group advantages or a critic. GRPO removes the need for a learned value baseline; it does not remove the need to score rollouts.

DPO avoids a separately deployed reward model during fitting. It does not remove overoptimization or poor preference coverage. Refreshing candidates changes collection, not automatically the update objective. DPO loss and derivation → and the coefficient convention → explain the exact objective.

Five roles, not necessarily five full copies

The old policy measures collection mismatch. The reference measures drift from a chosen anchor. These roles can coincide at initialization but diverge later. Adapter sharing, cached reference scores and stored behavior log probabilities can avoid five full model copies. Their identities and score contracts must still remain distinct.

Learn a preference score

Classic RLHF first trains an assistant with supervised fine-tuning. Human comparisons then train a reward model. A Bradley–Terry loss favors the chosen response when its score exceeds the rejected score.

RL then samples current answers and uses these scores as rewards, often with a reference penalty. The reward model can be exploited outside its training distribution. Keep held-out human or task evaluation distinct from the scalar the policy optimizes.

Preference Optimization → explains direct comparison fitting without the standard online reward-model loop. Data collection and the update objective are separate choices.

Writing quality or research synthesis often has no executable success test. Human comparisons, rubric judges, reference answers and AI-generated preferences can supply feedback instead. Reinforcement learning from AI feedback (RLAIF) replaces some human comparison labels with model-generated labels.

Constitutional AI uses principle-guided critiques and revisions, then AI preferences for RL. It is a feedback pipeline, not a new policy-gradient optimizer. Constitutional AI ↗

Inspect verbosity preference, style homogenization, annotator bias, reward-model overoptimization and distribution shift. Policy and judge can share correlated errors. Judge ensembles do not guarantee independence. Combine exact outcomes where available with explicitly scoped rubric scores elsewhere.

Terminal outcomes and intermediate feedback

An outcome reward evaluates the completed answer or task. Its meaning can be clear, but many decisions share one scalar. A process reward scores intermediate steps or actions and supplies more local feedback.

Denser feedback can reduce variance and identify repeated mistakes. It can also reward plausible-looking but unproductive behavior. A process reward model (PRM) is an evaluator, not proof that each step is mathematically correct. Final verifiers and process rubrics can disagree.

PRM800K and step-supervision work motivate evaluating reasoning steps, including candidate selection. A selection result does not establish that the same evaluator improves every training run. Preserve evaluator error and selection cost when comparing systems.

Make success difficult to counterfeit

RLVR works well when successful outcomes can be checked reliably. Examples include unit tests, symbolic equivalence, exact answers and environment completion. Learned judges remain useful where success is not executable, but their errors are part of the reward contract.

For repository tasks, check both fail-to-pass behavior and pass-to-pass regression tests. Construct hidden tests before exposing the task when possible. A generated solution and its generated verifier can share the same mistake; use independent inspection and adversarial checks.

Treat timeouts, malformed output, missing tests and grader crashes explicitly. An infrastructure failure is not necessarily a reasoning failure. Never default an exception to success.

Isolate network, hidden files, logs, caches, bytecode and future commit references. The model can optimize access to the answer rather than task completion. Audit high-reward samples with independent graders and include reward-hacking probes throughout training.

A verifier must canonicalize harmless answer differences without accepting different meanings. Parser ambiguities, hard-coded visible tests and malformed outputs with default success scores are concrete failure cases. Audit the checker before increasing rollout volume.

A rubric-based AI judge can cover research synthesis, ambiguous planning or creative work. It is not automatically an exact verifier. Softer feedback inherits evaluator misspecification; several judges can still share the same error. Budget submissions and keep hidden checks inaccessible through tools. Public checks should give diagnostic feedback without exposing the complete private grader.

What clipping actually changes

Reusing a rollout after an update makes it off-policy. Proximal Policy Optimization (PPO) compares each recorded action's current probability to its behavior probability. A critic commonly supplies advantages through generalized advantage estimation (GAE).

For A>0, the objective stops increasing beyond u. For A<0, it stops improving below l. On the adverse side, the ordinary ratio branch can still have a large derivative. Clipping does not force all ratios into [l,u] or guarantee a KL trust region.

With a critic, GAE combines temporal-difference residuals.

λ trades bias and variance. Keep terminal bootstrap, truncation and reward timing explicit. A censored response is not necessarily a true terminal state.

Critic-free methods have not universally replaced PPO. Open-Reasoner-Zero reports a useful vanilla PPO/GAE reasoning recipe, while VAPO revisits value-based optimization for long reasoning. These results motivate evaluating critic quality and cost, not declaring one family obsolete. Open-Reasoner-Zero ↗ · VAPO ↗

Compare responses to the same prompt

For one prompt, sample G responses and compute rewards R₁…R_G. Group Relative Policy Optimization (GRPO) estimates the baseline from these responses instead of training a value network.

With rewards [1,0,0,1], μ=.5 and σ=.5. The normalized advantages are approximately [+1,−1,−1,+1]. For outcome-only GRPO, every scored token of a response receives its response advantage.

Broadcasting an advantage does not locate the decisive token. Group contrast estimates whether a whole attempt was better than nearby attempts. The reward model or verifier can be learned or executable. DeepSeekMath section 4 ↗ introduces the original recipe.

Remove the rollout from its own baseline

A group mean includes the scored response itself. Under iid responses and an unnormalized centered reward, that self-inclusion scales the expected reward gradient by 1−1/G. It is not an independent action baseline.

The other sampled rewards are action-independent for response i under the iid assumptions. RLOO removes the self-inclusion scale effect without a learned critic. It does not solve sparse successes, judge errors or length weighting. It is a serious baseline, not a universally superior optimizer.

When a group supplies no contrast

If all outcomes are equal, group-centered task advantages vanish. All-pass prompts cannot indicate which successful response is better; all-fail prompts cannot distinguish failed strategies.

This does not imply that the optimizer always makes no update. KL, entropy, process rewards or another baseline can supply gradients. Weight decay can also change parameters. Record the fraction of zero-contrast groups separately from optimizer activity.

Filtering these prompts improves contrast per learner update, but changes the sampled task distribution. Keep their completed outcomes for curriculum and difficulty estimation before filtering.

Who receives the gradient budget?

The same token surrogate can allocate very different update mass depending on its denominator. First distinguish completion masks, token weighting and prompt weighting.

For lengths [2,8], response means assign each response mass .5. A valid-token mean assigns .2 and .8. A fixed Lmax=8 assigns .125 and .5. These are different objectives, not interchangeable implementation details.

Dr. GRPO removes response-length normalization and reward-std scaling in its recipe. DAPO uses valid-token normalization. Batch-dependent token denominators and group std can introduce their own dependence on difficulty and length. No reduction eliminates every bias.

Check the current trainer version. Current TRL main differs from historical DeepSeekMath defaults; a configuration name alone does not establish the equations.

Keep exploration and long responses useful

DAPO addresses four practical failures of long reasoning RL. These changes work together but have separate responsibilities.

A softened penalty does not make an unfinished answer successful. Distinguish reward shaping, loss masking and a hard generation limit. Record whether terminal reward was observed or censored. Selecting mixed groups can improve useful contrast while consuming many extra samples.

In the Smol Playbook’s reported /no_think experiment, naive GRPO increased response length despite the concise mode. An explicit overlong penalty addressed that behavior. This is a recipe-specific observation, not proof that shorter reasoning is always better. Smol behavioral-training context →.

Token, prefix and sequence likelihood ratios

A token ratio corrects the action distribution at a recorded prefix. It does not by itself change how likely the new policy is to reach that prefix. Whole-trajectory importance sampling includes every sampled action.

A cumulative ratio $\rho_{1:t}=\prod_{j=1}^{t}\rho_j$ corrects the prefix state/action distribution. For a function of that prefix, future ratios integrate out under matching environment dynamics and support. A sampled final reward is not generally prefix-measurable.

Consider two independent binary actions. Both policies choose the first action with probability 0.5. The target chooses the second with probability 0.8; the behavior policy uses 0.2. The reward is $R=a_1a_2$. The true gradient with respect to the first action’s Bernoulli logit is 0.2. Prefix-only weighting of the behavior terminal reward gives 0.05. The target conditional value $Q(a_1)=0.8a_1$ restores 0.2. The first-logit score is $a_1-0.5$. Thus the gradient is $0.5(1-0.5)\times0.8=0.2$ under the target. Using the behavior terminal reward instead gives $0.5(1-0.5)\times0.2=0.05$.

Accumulate ratios in log space. Even finite log sums can overflow after exponentiation. Clipping or truncation controls variance at the cost of changing the estimator; it is not an exact unbiased correction in general.

GSPO, CISPO and CTPO solve different problems

GSPO connects a response reward to a response-level update. Its reported MoE stability benefits are recipe-specific. It does not prove that all routing mismatch disappears. GSPO equations 5–10 ↗

MiniMax-M1 effectively disables the lower weight bound and tunes the upper bound. Its Qwen2.5-32B math comparison reports comparable DAPO performance with half the training steps. This is not a hardware-independent twofold speedup. MiniMax-M1 section 3.1 ↗

CTPO's practical normalized group outcome and clipping do not inherit the theorem for an exact prefix-dependent advantage. Its reported tool-math setup uses Qwen3-4B/14B, up to five turns,8k trajectory limit, batch 512×8 and LR 1e−6. Evaluation avg@32 averages success over 32 attempts; it is not pass@32 coverage. CTPO theory and setup ↗

Anchor behavior without confusing policy roles

A reference penalty discourages drift from a selected policy. It addresses a different question than the behavior ratio used to reuse rollouts.

Replayed actions from another policy do not automatically give that expectation. Use appropriate sampling correction or describe the surrogate being used. KL can enter shaped rewards or the objective; avoid adding it twice.

Some successful verifiable-reward recipes set $\beta=0$. That is not evidence that reference anchoring never helps or that general behavior stays unchanged. Monitor held-out general and safety behavior alongside the task reward.

Score the distribution that actually generated tokens

A rollout is sampled model behavior, from one response to a tool-using trajectory. On-policy samples match the policy being optimized; reused samples can come from an earlier behavior policy. Record the actual distribution used by the rollout engine. Temperature divides logits before normalization. Top-p sampling restricts support and renormalizes probabilities. A raw untempered full-vocabulary probability is not the probability that generated a truncated candidate.

For replay, preserve token IDs, template, temperature, candidate support, behavior log probabilities and policy versions. Scoring the same recorded support can align the update contract; it does not reconstruct missing actions outside that support or eliminate prefix-distribution mismatch.

Numerical precision, quantization, fused kernels and expert routes can change behavior likelihoods. Routing replay preserves selected expert indices for the scored execution where required. Treat rollout/training alignment as a numerical and data contract, not merely a weight synchronization call. MoE → connects routing to RL stability.

Control diversity without assuming a guarantee

A model can improve reward by concentrating probability until useful alternatives disappear. Token entropy measures uncertainty at sampled states, not complete task coverage.

Each row starts from the same k. These are controller arithmetic examples, not measured entropy trajectories. Saturation at kmax prevents further increases. A fixed entropy bonus instead adds a gradient term; its useful coefficient can change with reward scale and training stage.

Increasing u changes which positive-advantage updates remain active. It does not mathematically guarantee higher entropy after the next shared-parameter step. Delayed measurements, saturated controls and changing task mixes affect feedback.

A separate hard outer ratio clip can limit extreme ratio magnitudes. Detached weighting and differentiated clipping have different gradients. Log the actual masks and active bounds instead of describing all variants as the same PPO trust region.

Why a binary signal can still teach

Dense token prediction provides a target at every observed position. A terminal verifier may return one scalar after thousands of autoregressive steps. Counting external labels per generated token exposes a real bandwidth and generation-cost difference. It does not measure usefulness for a specific downstream task.

Likewise, token entropy is at most log₂V for vocabulary size V. An individual very unlikely token can have much greater surprisal. Do not confuse these average and event-specific quantities.

A pretrained policy already defines a structured search distribution. A terminal score selects complete trajectories from that prior. A verifier, environment and task generator also encode information; their construction is not free evidence.

The reward multiplies a structured score gradient. Sampled actions, prefixes, policy probabilities, model derivatives and the baseline supply its direction. One bit can select a high-dimensional update without encoding an arbitrary missing strategy. A successful trace can reinforce decisive steps and incidental wording together; failed traces may contain useful partial work.

An update can align with a narrow verifier and remain a noisy estimate. Reducing learning rate changes parameter displacement, not the information in one sample. Batching independent estimates reduces variance; momentum can aggregate estimates over time. None of these proves that most pretraining gradients are statistically noise.

Here p is per-attempt success probability. Practical support means useful behavior is reachable within the rollout budget, not merely assigned positive softmax probability.

For rare successes, 1−(1−p)^K≈Kp requires Kp to remain small. At a larger budget, the probability saturates below one rather than growing linearly forever. Dense training, tools, a changed task mixture or better sampling may be needed before policy-gradient updates can usefully select successes.

Pretraining supplies language, knowledge, representations and action primitives. Targeted SFT or mid-training can raise initial success and reduce the policy shift needed under limited updates, KL/clipping and rollout budgets. Exploration still searches decompositions, code edits, tool calls and recovery strategies. New combinations need not have appeared verbatim in demonstrations.

The prompt distribution, task generator, compiler, tests, tools, environment dynamics and curriculum contain reusable information too. Count these external artifacts when comparing information budgets. A rich verifier can still expose only pass/fail for one attempted trace.

Entropy and information accounting →

Successful traces are a conditional distribution

Success-conditioned sampling changes the distribution. If successful traces are retained, their distribution is proportional to the old policy times the success indicator. Imitating that subset can therefore produce a nonzero learning signal.

Rejection-sampled SFT can be a useful baseline. It ignores some failed-attempt information, may narrow diversity and can overfit a fixed selected set. RL can refresh candidates and use signed advantages; this does not establish universal superiority over imitation.

Dense teacher log-probability feedback is another way to transfer useful behavior. Keep student/on-policy distribution and teacher identity explicit. The Distillation reader explains teacher feedback on student-generated histories. Scalar reward bandwidth alone does not determine adapter capacity; compare LoRA rank, target layers and optimization at matched budgets.

On-policy teacher feedback → · LoRA and full-parameter adaptation →

A successful policy can be distilled rather than rediscovered by search. Dense teacher feedback is valuable when it targets useful behavior; more token feedback does not by itself prove better alignment with external correctness. Compare diversity, filtering, sequence length, feedback quality and refresh cadence, not only label count.

On-policy teacher transfer can consolidate specialists, teach smaller students or restore assistant behavior after domain adaptation. The student supplies the history; the teacher scores those same actions. Teacher-forced scoring avoids generating a second teacher trace, but still costs compute.

Immediate sampled-token log-ratio feedback gives a fixed-prefix reverse-KL gradient under a matching sampling contract. It is not generally the complete sequence-KL gradient: early actions change later prefixes. The conditional derivative → and future visitation credit → retain the derivations. Behavior restoration → is separate from adding facts absent from the teacher.

Accuracy can improve while coverage shrinks

Evaluate both the chance of one successful attempt and the range of tasks reachable with more attempts. These can move in opposite directions.

For two equally weighted tasks, probabilities [.4,.4] yield mean pass@1=.4 and pass@10≈.994. Probabilities [.9,0] yield mean pass@1=.45 but pass@10≈.5. The better single-attempt policy lost the second task in this constructed example.

This example is not a measured model comparison. Finite failures cannot distinguish zero success probability from very small probability. Coverage also requires recognizing the successful candidate; judge error and selection cost matter.

Measure the remaining value of more attempts

Sharpening Tax measures lost sampling scalability after post-training. The official implementation estimates coverage and all-attempt success separately from n recorded attempts with c successes.

Estimate pass@k for each task, then average these coverage curves over the benchmark. Compute A and S from that dataset curve, as in the paper and official metric implementation. Averaging separately normalized per-task S values is a different metric. Use paired task bootstrap intervals. A perfect pass@1 leaves zero failure headroom, so normalized S needs an explicit convention instead of blind division.

For two equally weighted tasks with success probabilities .2 and .8, the K=2 dataset curve is [.5,.66]. Its A=.16 and S=.32. Separately normalized task values are .2 and .8, whose mean .5 is not the reported dataset S.

Official metric implementation ↗

The paper compares 14 checkpoint pairs on BFCL, WebShop and ACEBench. Exact post-training recipes are unknown; base and final policies use different scaffold conditions, and attempts need not have equal compute. The observational comparison does not show that RL cannot learn capabilities.

Sharpening Tax paper ↗ · Official metric and controller ↗

Adjust temperature to estimated task difficulty

Posterior-Tempered Group Sampling (PTGS) allocates exploration temperature per recurring task. It keeps discounted successes s and failures f. The current target success rate q defines a Beta prior and a temperature pivot.

With $q=0.5$, $\tau=1.5$ and $T_{\mathrm{ref}}=1$, draws 0.1, 0.5 and 0.9 produce temperatures about 1.38, 1 and 0.72. This pivot does not force actual success probability to q. Temperature helps only if useful trajectories become more likely.

Carry the temperature through the complete loop

Advance the target schedule before drawing temperatures. Use a stable recurring task ID, not changing prompt text. Draw one temperature before dispatch and retain it for every response and turn in that group. Update the posterior from completed outcomes before removing all-equal groups.

temperature = controller.draw(task_id)
trajectory = rollout(policy, temperature=temperature)
controller.observe(task_id, successes, completed_count)
if keep_group(trajectory):
    batch = attach_metadata(trajectory, temperature)
    train_logps = score(policy, batch, temperature=batch.temperature)

Require Tref>0 and 0≤γ≤1. γ=1 retains all previous evidence; smaller γ discounts it. Checkpoint counts, target schedule and random-number-generator state. A nonrepeating seed stream does not accumulate per-task history. A learned difficulty estimator for unseen tasks would be an extension requiring validation.

Keep validation at its declared fixed evaluation temperature. Treat a missing temperature field at the actor as an error. Monitor fallback rows: nonzero counts indicate mismatched group IDs. Count one terminal outcome per trajectory, not one per turn. Compare entropy measurements at a common temperature across experiment arms.

For asynchronous collection, store the assigned temperature in dispatch metadata. Recomputing it after posterior updates describes a different behavior distribution.

The reported Qwen2.5-7B-Instruct Sokoban PPO example improves pass@1 46.5→61.1 and pass@128 55→69.7, averaged over five runs. The study also tests FrozenLake and both PPO and GRPO. It is not a universal temperature result.

The released repository provides metric and controller examples, not a complete benchmark reproduction. Compare coverage and accuracy at matched tasks and settings, with generated tokens, actions, accelerator time, wall-clock time and selection cost. Use intermediate SFT, RL and distillation checkpoints to attribute changes to a stage.

What reasoning experiments do and do not prove

R1-Zero shows that a base model can acquire useful reasoning behavior through a particular RLVR pipeline without an initial reasoning SFT stage. Readability, mixed languages and general alignment remain separate problems. Hybrid cold-start/SFT/RL stages address these in other recipes.

Studies of base versus trained sampling support ask whether RL discovers new successful behavior or concentrates existing behavior. A finite evaluation cannot fully identify support, and changing scaffolds can change the task. Distinguish measured success, the authors' mechanism hypothesis and a universal impossibility claim.

Spurious-reward findings show that some indirect signals can correlate with improvement in particular setups. They do not make unreliable reward design harmless. Test clean held-out verifiers, budgets and intermediate checkpoints rather than attributing every base-to-final change to RL alone.

Outcome feedback leaves the trace unspecified. Decomposition, backtracking, intermediate checks, self-written tests and adaptive stopping may emerge or become more frequent. Their appearance does not prove that the base model had no related behavior. These pipelines are alternatives, not a universal ranking.

Increase rollout counts or generation budgets, mix easier tasks, or initialize from useful demonstrations when successful contrast is too rare. Do not maximize difficulty for its own sake. All-equal group filtering applies to centered group objectives, not every possible RL update. Penalizing length before reliable success can suppress exploration. Recheck knowledge, instruction following and open-domain usefulness after specialized math/code stages.

The bandwidth/alignment/prior explanation is a framework, not a proven general theory. It suggests rapid narrow gains when probability allocation changes, uneven domain improvements, cold-start failures and sensitivity to an optimized reward proxy. These predictions do not identify the cause of every observed gain.

Classical RL contains on-policy and off-policy families. Replay is not universally safe there or universally invalid for LLMs. Long-trajectory ratio products already explain one concrete mismatch risk; clipping or truncation changes the correction. A pretrained-policy refinement problem differs from jointly learning representations and actions from a weak prior.

Before scaling, measure per-prompt pass@k, reward entropy, group contrast and held-out general behavior. Audit reward integrity and sampling provenance. Compare RL, rejection SFT and teacher feedback with matched rollout tokens and accelerator time. Dense process feedback is useful only when its added credit outweighs evaluator bias and exploitability.

Spurious-reward experiments improved some Qwen math policies under random, format-only or incorrect feedback. Comparable signals did not consistently help Llama or OLMo. The paper relates these gains to model-dependent prior behavior and clipping bias. This is not permission to replace task verification with random labels. Spurious Rewards ↗

Compare real rewards with spurious baselines across multiple model families. Improved pass@1 can be economically useful without proving new support. For missing facts, compare continued pretraining, demonstrations, retrieval and teacher transfer instead of assuming outcome RL is the most efficient source of knowledge.

Decouple collection from learner updates

A synchronous collector waits for its slowest required rollout before the learner updates. Agent task durations vary widely, so useful GPUs can wait for long environment tails. Asynchronous systems overlap collection, grading and optimization.

A longest-tail versus completion-quantile estimate explains possible utilization gains. It is not an exact wall-clock prediction: synchronization, grading, batching and communication remain. AReaL's reported up-to-2.57× result is source-conditioned, not a portable promise.

This example changes the collection threshold from 9 s to 3 s, not total training time. Unfinished attempts keep their trajectory state and resume later. A GRPO learner must still respect same-prompt group membership and an explicit incomplete-group policy. It must not combine unrelated prompts merely because their attempts finished together.

Generation can consume more hardware than learning when traces are long or each prompt needs many candidates. Tool latency adds another resource bottleneck. Measure rollout, learner, grading and transfer costs rather than assuming inference dominates every run.

A partial rollout can span policy versions

A completed rollout may come from an earlier policy. Measure its age in learner updates, not just elapsed seconds. A partial rollout can pause, then continue after a new checkpoint arrives.

Treating the entire trajectory as v10 is false after a v12 continuation. Routing/cache reuse needs a declared validity contract. Some systems instead enforce internally version-consistent trajectories, as in DORA; that is a different design choice.

Policy-age limits, likelihood-ratio weighting and stale-token masks trade sample efficiency against bias and stability. They do not guarantee that a changed suffix distribution is harmless. Preserve segment provenance and task outcomes even when some tokens are excluded from the learner.

INTELLECT-3: mask ratio tails, not just stale versions

A version limit bounds policy age. It does not bound probability mismatch. INTELLECT-3 reports mismatch even with identical trainer and inference weights. Its in-flight updates can also mix policy versions within one trajectory.

The denominator must describe the distribution that sampled each token. A checkpoint name alone cannot reconstruct that distribution after temperature, candidate filtering, numerical execution or policy versions change.

A ratio of six contributes zero through this masked term; it does not become five. This differs from clipping a ratio to an endpoint. It also differs from PPO's signed minimum surrogate. The table shows coefficients, not the complete gradient.

The report uses mean-centered rewards without group-standard-deviation normalization in this expression. Equal group rewards therefore give zero advantages. Masking ratio tails intentionally changes the estimator; do not call the masked objective unbiased importance sampling.

Record behavior log probabilities and segment versions before filtering. Monitor rejection by task and length. Aggressive filtering can remove a whole capability from the accepted batch while token throughput stays high. Reference-policy KL is a separate behavioral anchor.

INTELLECT-3 refills completed rollout-group slots and uses round-robin requests across inference clients. Client balancing is not a harness-family mixing policy.

INTELLECT-3 §§2.1.2–2.1.5 ↗ · §3.3: masked importance sampling ↗ · Explicit harness-family weights →

Do not let fast tasks choose the curriculum

The fastest harness can dominate if the learner consumes whichever samples finish first. This changes the task mixture even when dispatch quotas were equal. Completion bias is distinct from policy staleness.

Track dispatch, completed groups, accepted groups, valid tokens and objective coefficient mass by source. Equal task completion counts do not imply equal gradient contribution. Long trajectories, filtered groups, reward scale and loss reduction change the contribution.

A harness also defines prompts, context management, action format, tool semantics and environment behavior. Check these contracts against deployment. A simplified tool API or clean simulator can reward behavior that does not transfer. Matching one production harness is not a guarantee of cross-harness generalization; evaluate held-out variants.

A quota or deficit mixer can target harness families or task sources. Weighting families is a coherent experimental choice, not proof of a better curriculum. Define whether weights act on dispatch probability, accepted prompt counts, token mass or the objective. Do not accidentally apply the intended weight twice.

Keep the task alive when the GPU worker stops

An environment session should outlive a replaceable GPU rollout worker. Preserve files, tools, process state and task identity so interrupted reasoning can resume without restarting the world.

Pause/resume can release resources while a tool call waits. Fork the environment for judging when grading would mutate it. The judge should inspect a clone rather than changing the agent's task state.

AgentENV's reported Firecracker incremental pause/resume costs are about 133ms/49ms under its measurement conditions. Treat these as reported infrastructure timings, not universal VM costs. Isolation and hidden-verifier security remain required even with rapid snapshots.

AgentENV uses incremental dirty-page checkpoints to suspend mutable microVM state, release runtime resources and restore before a later tool action. The reported 133 ms checkpoint / 49 ms resume values do not include every workload's complete recovery cost.

Separate judge forks allow independent checks and reproducible failure inspection. Grading must not mutate the agent's authoritative final state.

Store the complete reusable state

Keep active inference state near the GPU, move idle prefixes to CPU when useful, and offload weights or optimizer state under a measured budget. Moving data is not free; include transfer, restore and eviction costs.

A hybrid attention/recurrent model needs more than token KV. At a reusable boundary, preserve every recurrent state plus required attention cache. Sparse checkpoints limit which prefixes can be restored exactly. A cache hit with incomplete state is not correct reuse.

Colocating rollout and learner workers can reuse accelerator capacity and avoid a permanently idle fleet. Dedicated fleets can reduce interference. The useful choice depends on rollout/train ratios, sequence lengths, memory pressure and weight-transfer costs. Parallelism → explains layout/resharding.

The report describes colocating phases on a few hundred GPUs. This saves a permanently separate fleet but adds offload and phase-transition costs. Only evicted idle prefixes need write-back; copying active state unnecessarily adds traffic. Compare total useful accelerator time, not each phase's isolated peak.

Optimize useful progress, not queue length

Bound queues by GPU memory, CPU cache, environment capacity and grading resources. Admission control should consider the predicted state size, not only active request count.

Publish checkpoints atomically with version and inference layout. Validate dtype/quantization, expert placement and templates before accepting the version. A learner-format checkpoint is not automatically an inference-ready shard set.

Use watchdogs, timeouts and recoverable task snapshots. Retrying a failed worker must not silently duplicate training on the same accepted sample. Distinguish task failure, infra failure, grading failure and abandoned work.

Measure useful accepted samples per accelerator-hour and held-out progress per total cost. Raw token throughput can rise while completions become stale, biased or unrecoverable.

Weight refresh can require resharding, layout conversion, dtype conversion and replication management. Sparse refreshes increase policy age; frequent refreshes consume transfer, reload or requantization time. Tune cadence alongside the update recipe and inspect reward/completion/ratio tails by staleness bucket.

Learner and rollout precision need not be bitwise identical. Measure the mismatch, keep its scoring contract stable, and validate any engine or precision transition. Preserve tokenization, assistant rendering, tool-call formatting and turn boundaries; protocol changes can alter a multi-turn task.

Use replica and progress watchdogs, liveness checks and router-level circuit breaking. An isolated inference replica failure should reduce capacity rather than terminate the whole job. Recovery must preserve sample identity to avoid duplicate accepted updates.

Throttle admission when active/queued requests, GPU KV use, CPU prefix use or environment capacity exceed the budget. Raise it only when queues and memory permit. Report restore latency, failed grading and reward/completion rates by policy age, not just average throughput.

MAI: repeated RL climbs

MAI Thinking-1 treats post-training as repeated climbs rather than one final alignment pass. Its GRPO-style loop uses response-normalized updates, group filtering, adaptive upper clipping and a hard outer ratio limit. Top-p replay aligns the candidate support used in scoring.

Entropy control responds to observed concentration. It is not an additional correctness verifier or a proof of bounded KL. Keep the group filter, reward scale, clip controller and outer weight cap separate when reproducing the recipe.

Self-distillation periodically consolidates selected late-climb behavior, with roughly one million traces in the described selection. Prompt diversity matters more than repeatedly sampling a small prompt set. Short-context consolidation can forget long-context behavior, so evaluation and recovery must cover both.

Distillation changes the policy's training distribution and restart point. A final checkpoint's gain cannot be attributed to GRPO alone without intermediate ablations.

MAI Thinking-1 recipe →

GLM-5: reasoning and asynchronous agents

GLM-5 separates its reasoning and agentic workloads. The disclosed reasoning recipe uses groups of 32, prompt batch 32, binary math/science/code/tool rewards and no KL penalty. Lower/upper clipping parameters are .20/.28, with an IcePop likelihood cutoff 2.

These values describe one report's implementation. A likelihood cutoff, signed PPO clipping and KL regularization are different interventions. Resetting an optimizer after an inference-engine transition also changes the operational recipe.

The Slime infrastructure runs more than 1,000 concurrent rollouts. Token-in/token-out (TITO) preserves exact generated IDs and metadata rather than retokenizing displayed text. Double-sided importance treatment, policy-age checks and invalid-sandbox filtering determine what reaches the learner.

For incomplete groups, the system pads only when the surviving group is sufficiently complete; otherwise it drops the group. Such handling must retain masks and avoid treating missing scores as real failures. The report also uses rollout identity in KV hashing and FP8/MTP inference.

Agent data include more than 10k software tasks across nine languages, thousands of Docker/Harbor environments and reported build success above 90%. A two-million-page web graph supplies browsing tasks. Visual artifacts require static, runtime and perceptual checks; one binary test does not capture every modality.

A subsequent teacher-feedback phase uses student prefixes and token log gaps. This is connected to RL but is a distinct distillation signal, not evidence that all preceding stages use the same advantage.

GLM-5 recipe →

Kimi K3: effort and specialist consolidation

Kimi K3 uses SFT, specialized domain/effort training and consolidation through nine specialist teachers. Its MOPD stage combines task reward with clipped stop-gradient teacher–student token log-probability gaps on student-generated context.

The effort controller starts with a problem-specific cold-start budget $b_0(x)$. A response exceeding the selected budget $\tau b_0(x)$ receives the described length consequence $R=-1$. Lower-effort multipliers anneal down; the max-effort mode retains more exploration room.

For agents, the budget counts cumulative model outputs and tool arguments across turns. It is not only the final hidden reasoning length. A short last answer can still follow an expensive or cyclic trajectory.

The agent task definition combines initial state, goal, tool access, budget and verifier. Modular harnesses supply prompts, context management, skills, memory and subagents. Tasks need not supply a reference trace; private grading and public diagnostic feedback serve different purposes.

Partial rollout workers preserve the environment and resume with newer weights. Such trajectories require per-segment behavior provenance. The report's 51M sandboxes and 1.5M images describe accumulated infrastructure scale, not necessarily simultaneous concurrency.

Kimi K3 teacher and effort recipe →

DeepSeek V4.1: constructed tasks and shared workers

DeepSeek V4.1 Flash follows SFT→RL→final on-policy distillation, drawing on more than 40 teachers. Teacher mixing weights and several RL hyperparameters are undisclosed; do not fill these gaps with standard GRPO defaults.

Task constructors learn useful difficulty and correctness. Separate planners/builders, independent solvers, inspectors and repair steps construct environments and tests. Coding evaluation combines new fail-to-pass checks with regression pass-to-pass checks.

Mock services can turn real workflows into controllable tasks. Their fidelity needs validation: an agent can exploit a simplified simulator even when it passes that simulator's grader. Multiple scaffold versions include Claude Code, OpenCode, Pi and DeepSeek's own harness.

Sample-level dispatch fills learner groups without waiting for every earlier group. Do not combine unrelated prompts under one group baseline. Shared GPU workers can be preempted at token boundaries and reuse selected cache/expert states under a version contract.

The system masks stale tokens under its criteria; thresholds are not fully disclosed. Completion-time bias toward short tasks needs mixture control in addition to staleness handling. A mask protects selected updates but does not make the entire mixed-version trajectory on-policy.

The report maps effort to a 1–100 scale and API low/high/max values 50/75/100. A larger effort value reduces length pressure under this formula. Reproducing it needs the undisclosed scale constants and the same definition of counted tokens.

DSec reports live-container density increasing from roughly 1,000 to over 2,500 per node. These are not simultaneous builds. NUMA-local workers, low-priority SCHED_IDLE placement, AppArmor/eBPF and isolated networking support predictable resource use. Sandbox crashes must become explicit failed signals, not accidental success.

DeepSeek V4.1 agent and RL recipe →

MiMo: reward groups, masked gradients and mixing

MiMo V2.6 makes the entire RL loop a design subject: reward construction, group quality, token weighting, rollout scheduling and arithmetic alignment. It is not accurately summarized as vanilla GRPO.

Groupwise Rubric Synthesis (GRS) builds offline solution and behavior rubrics, then reuses them online. Reward combines the test outcome with rubric-based quality. In contrast, Groupwise Advantage Redistribution (GAR) ranks passing trajectories within an online group.

The GAR grader considers precision, minimality, side effects and craftsmanship. Confirmed hacks receive zero before group statistics are recomputed. Passing positive advantage is redistributed according to quality factors fᵢ∈(0,1].

Under those assumptions, GAR changes allocation without creating positive mass. Soft reward cases require an explicit eligibility/fallback rule. Its ablation reports restrained turn/token growth with rising success, not a theorem that grading every group is worthwhile.

The mean-per-response choice prevents long trajectories from dominating. That goal differs from DAPO's equal-token allocation. Four sign-specific lower/upper bounds begin around [.2,5] and adapt with entropy. Positive- and negative-advantage sides have different restrictions.

On positive trajectories, flagged tokens are masked and their positive mass moves to clean tokens. On negative trajectories, flagged spans receive more blame while preserving the specified mass. Infrastructure failures can be removed. Masks, redistribution and reward rejection must not be conflated.

Four harnesses improve the reported mean from roughly 50 to 66 under the held-out harness evaluation. Equal completed task counts would still not imply equal token or gradient mass. Across 25 sources, mean generated tokens vary 90× and active durations 66×.

Adaptive concurrency, a deficit-aware mixer, predicted KV admission and limited replay protect the intended mixture. Sample→Sequence→Context→Segment records branches, compaction and selective loss. Metadata scheduling is separate from bulk tokens, images, probabilities and expert routes.

MXFP4 quantize/dequantize after each update aligns arithmetic. R3 replays expert selections, and top-p candidate replay preserves sampling support before renormalization. These interventions address different sources of behavior/training mismatch.

Reward integrity includes cleanup of logs, caches, bytecode and future commit references, network isolation, hacking agents and offline/online audits. Reported confirmed hacks below 2% are conditioned on detection, not proof of zero undetected exploits.

MiMo V2.6 full recipe →

Nanbeige: match feedback to compact-model competence

Nanbeige 4 separates STEM, practical coding and human-preference stages. Domains learn at different rates, so a permanent fixed mixture can leave one capability behind. Each stage uses on-policy GRPO, no KL penalty and truncated-sequence loss masking.

Before each stage, the latest checkpoint samples 16 responses per question. Retain only .1<avg@16<.9. With integer binary counts, this means 2–14 successes out of 16. Recompute difficulty because transfer and training change which tasks supply contrast.

A Python/symbolic verifier checks equivalent STEM answers. Coding tasks can be constructed backward from a valid solution and executable public/private tests to a problem statement. Independent validation is still necessary when generated artifacts share mistakes.

Nanbeige 4.2 addresses cyclic reflection, delayed termination and malformed continuations with two RLHF stages. Think-mode reward training is followed mainly by Non-Think general-response training. All following checkpoints are evaluated in Think mode.

These observations support cross-mode behavioral transfer. They do not isolate every reward component or prove that shorter reasoning always improves accuracy.

Correct short traces set a problem-specific budget. The penalty increases when the policy already succeeds reliably; harder tasks retain exploration room. Constrained phases alternate with free-expansion phases rather than making short outputs universally preferable.

Agentic training combines final outcomes with action-centric tool accuracy, information gain, progress and recurring-error rubrics. Compact-model agent tasks favor shorter trajectories and relatively high pass@8. This differs from selecting mixed-outcome reasoning tasks near the frontier.

The report shows normalized single-turn error falling about 20%, with validation rising 66→peak 71. These are reported quantities with different meanings, not one shared success percentage.

Nanbeige 4 staged RL and full recipe →

Nanbeige 4.2 behavioral training and full recipe →

Qwen 2.5: keep the reward model distinction

Qwen 2.5's original note incorrectly says GRPO removes the reward model. The primary distinction is that group baselines remove the separately trained critic. Learned reward models can still provide scores.

The report uses SFT, offline DPO and online GRPO. Its Online Merging Optimizer belongs to the offline DPO stage, not the online GRPO stage. Online learning uses reward-model feedback; math-specific reward-model usage must not be generalized to every helpfulness task.

Use the verified Frontier Recipes discussion for the full staged context. The broader lesson is to trace each label to an equation, feedback source and disclosed stage, rather than importing a newer Qwen recipe into an older report.

Frontier Model Training Recipes →

Qwen2.5 data,context and alignment recipe →

Connect collection, optimization and consolidation

A full run alternates collection with optimization. This conceptual sequence is not a universal recipe. Publish a learner version only after its update is complete.

Qwen3 reports long-CoT cold start, reasoning RL, thinking/non-thinking mode fusion and general-domain RL. Kimi K2 extends training into agent environments with outcome and softer feedback. These are stage choices, not requirements for every run. Qwen3 post-training ↗ · Kimi K2 ↗

Staged run design → · Agent tasks and environment synthesis →

Read optimization and systems signals together

Inspect sampled high-reward traces regularly. Retain evaluator versions, sampling budgets and intermediate checkpoints. A mean reward curve alone cannot establish stable reasoning, broad coverage or a healthy infrastructure pipeline.

Break reward down by prompt difficulty. Record invalid formats, verifier errors, rollout tokens per second and useful samples per accelerator-hour. KL to the reference and mismatch to the behavior policy measure different changes. Reward rising while held-out quality falls warrants checking reward hacking and overoptimization. One curve does not establish either cause.

Compact diagnostic questions

A small masked update, with an explicit contract

This conceptual PPO/GRPO kernel uses matrices [G,T] of sampled-action log probabilities. Masks exclude prompt, padding and declared censored spans. One response advantage broadcasts along T. Score helpers must shift targets exactly once.

adv = group_advantages(rewards).detach()       # [G]
ratio = (new_logp - behavior_logp).exp()       # [G,T]
a = adv[:, None]
raw = ratio * a
bounded = ratio.clamp(lower, upper) * a
surrogate = minimum(raw, bounded)
lengths = mask.sum(-1).clamp_min(1)
response_mean = (surrogate * mask).sum(-1) / lengths
loss = -response_mean.mean()                 # original response reduction
loss.backward()

This omits KL, entropy, distributed reduction and optimizer steps deliberately. Exclude empty responses under an explicit policy; clamping a denominator is not evidence that an empty response is valid. CISPO instead detaches its clipped weight and multiplies log probability. Do not change gradients by copying the same kernel under another name.

Before scaling, compare exact probabilities, masks and gradients on a tiny batch. Test all-equal rewards, signed clip boundaries, variable lengths, truncated outputs, stale versions and one resumed tool trajectory.

Sources and connected topics

PPO ↗ · DeepSeekMath/GRPO ↗ · RLOO ↗ · DAPO ↗ · Dr. GRPO ↗ · GSPO ↗ · CISPO ↗ · CTPO ↗ · Sharpening Tax/PTGS ↗

Raschka:reasoning-model training survey ↗ · DeepSeek-R1 ↗ · DeepSeekMath ↗ · Kimi k1.5 ↗ · Kimi K3 infrastructure ↗

Agent tasks and environment construction → · Serving and prefix reuse →

Millidge:the motivating framework ↗ · Ord:RL inefficiency ↗ · Patel:bits per sample ↗ · Williams:REINFORCE ↗ · Ross et al.:imitation and structured prediction ↗

InstructGPT ↗ · DPO ↗ · RLOO ↗ · DeepSeekMath ↗ · DeepSeek-R1 ↗ · Kimi k1.5 ↗ · Dr. GRPO ↗ · DAPO ↗ · Open-Reasoner-Zero ↗ · VAPO ↗ · GSPO ↗ · CISPO ↗ · CTPO ↗ · Process supervision ↗ · Constitutional AI ↗ · Base-model reasoning support ↗ · Spurious Rewards ↗ · HybridFlow / verl ↗ · AReaL ↗ · DORA ↗ · Qwen3 stages ↗ · Kimi K2 ↗

Sharpening Tax paper ↗ · Metric documentation ↗ · PTGS controller ↗ · Integration hooks ↗ · PTGS guide ↗ · Official repository ↗

Losses and Objectives → · Preference Optimization → · SFT → · Distillation → · Frontier Recipes → · Compact Recipes → · Decoding →

The interactive scenes are exact constructed examples, not measured training curves. Report observations retain their own task, model, harness and compute conditions.