Diffusion and Score-Based Generation

What does the generator learn?

A diffusion model does not learn to recover every original image from arbitrary noise. That recovery is impossible when corruption removes information. It learns conditional predictions that support sampling from a data distribution.

The fixed forward process creates training examples at known noise levels. The model learns a local denoising field. A reverse sampler then uses that field repeatedly to transform a simple prior into structured samples.

This distinction explains both the strength and the cost of diffusion. Local targets are available without solving the whole generation problem during training. Generation still requires a sequence of model evaluations. Acceleration can change the numerical solver, the learned trajectory map, or the generator objective. These are different interventions.

Start with Gaussian diffusion. The broader corruption view comes later, after the equations have a clear domain. This note owns diffusion-specific sampling and distillation; Distillation owns the general teacher–student framework.

Keep time, variance and state separate

For variance-preserving marginals, $a_t^2+\sigma_t^2=1$. Additive-noise examples instead use $a_t=1$. Do not transfer formulas between conventions without the signal coefficient. Batch inputs usually have shape [B,C,H,W] for images or [B,N,D] for latent patches. Broadcast one sampled time per example across its data coordinates.

Construct the forward Markov chain

Each transition reduces the previous signal and adds new Gaussian noise. The conditional dependence is Markov: the next state depends on the current state, not the complete earlier sequence.

For one coordinate, the conditional mean follows the signal multiplier. The accumulated variance satisfies $v_t=\alpha_t v_{t-1}+\beta_t$ with $v_0=0$. The recurrence gives $v_t=1-\bar\alpha_t$. This algebra is the reason that direct training samples are available; it is not an approximation that requires tiny steps.

The noise variables at different Markov transitions are independent. A visualization that reuses one ε across levels displays marginal couplings, not an actual realization of this Markov chain.

Sample any training noise level directly

Training samples a clean datum, a noise level, and ε. The formula creates xₜ in one operation. It does not simulate x₁ through xₜ₋₁. This makes the training target inexpensive even when the generation schedule has many levels.

For a constant $\beta=0.08$, after ten steps $\bar\alpha=0.92^{10}\approx0.4344$. With x₀=2, the conditional mean is about 1.318 and the conditional variance about .5656. These are distribution moments. A single noisy draw also includes its particular ε.

Store or compute the schedule at sufficient precision. Summing log α avoids losing small products; using expm1 can accurately recover 1−ᾱ near zero noise. The exact implementation still needs its schedule endpoints and broadcasting rules.

Watch signal disappear without confusing paths

The five dots use assigned standardized positions, not sampled images. Moving the stage changes the conditional Gaussian centered around one clean value. The figure explains signal contraction and variance growth without pretending to train or run a neural network.

Read the coefficients separately. The noise standard deviation approaches one, while the signal coefficient approaches zero. The mean reaches zero only in the limiting schedule shown by the formula, not at every finite terminal level.

When is terminal noise a valid prior?

Generation starts from a distribution we can sample. In a conventional variance-preserving schedule, the chosen prior is N(0,I). The forward terminal distribution approaches that prior when ᾱT is sufficiently small and residual scaled data structure is negligible.

A finite schedule with ᾱT>0 still contains signal. The marginal q(xT) is generally a mixture over clean data, not exactly N(0,I). If the terminal signal is large, training and generation begin from different distributions.

There are two separate checks: the schedule must reach the intended noise regime, and the reverse model must handle states drawn from its inference prior. Increasing the number of steps alone does not establish either check. A zero-terminal-signal construction needs endpoint-safe parameterization.

Reverse with the clean datum: a tractable target

Multiply the Gaussian likelihood of xₜ given xₜ₋₁ by the Gaussian marginal of xₜ₋₁ given x₀. Complete the square in xₜ₋₁. The precisions add; the precision-weighted means determine μ̃ₜ.

The formula is not yet a generator. At inference, x₀ is unknown. Its role is to supply a training target and a tractable variational comparison. At t=1, the conditional posterior knows x₀ exactly and $\widetilde\beta_1=0$; the model decoder remains a separate distributional choice.

Do not replace the unknown marginal reverse conditional q(xₜ₋₁|xₜ) with this clean-conditioned posterior. Integrating over possible x₀ values can produce a non-Gaussian mixture. DDPM, equations 6–7 ↗ gives the posterior.

Remove the clean datum from the reverse model

The model supplies a mean and either fixed or learned variance. Small-step reverse Gaussian approximations are useful, but a coarse step can cross an ambiguous or multimodal region. A Gaussian parameterization does not make that ambiguity disappear.

The network predicts noise, but the sampler converts that prediction into a reverse mean. Randomness is a separate operation. The final decoder convention and whether noise is added on the last step must be explicit.

Why noise prediction trains a denoiser

The injected ε is known, so the target is available. The model cannot infer the particular noise perfectly from an ambiguous xₜ. Under squared error, its population optimum is E[ε|xₜ,t], not the unknowable noise realization in every training pair.

That conditional average is sufficient for the corresponding marginal score. A denoiser is therefore useful even when its direct clean prediction would average multiple possible images. Iterative generation uses the whole field and its evolution, not one mean estimate at maximum noise.

Reduction over coordinates changes the scale of the gradient. Averaging versus summing, time weights, and the sampling distribution over t are part of the loss contract. Report them before comparing two training runs.

The VAE view and the variational bound

The forward chain is a fixed approximate inference process. The reverse chain is a learned latent-variable model. This resembles a VAE with many latent layers indexed by noise level. The autoregressive structure runs across levels, not necessarily pixels or text tokens.

For a fixed reverse variance, an intermediate Gaussian KL becomes a weighted squared error between posterior and model means. Substituting the noise parameterization gives a time-dependent weight on noise-prediction error. The simplified unweighted noise loss changes those weights. It is not numerically identical to the complete negative ELBO.

The latent-variable interpretation connects diffusion to variational inference. It does not imply that a learned reverse chain has competitive exact likelihood whenever it produces attractive samples. Information Theory separates coding objectives from sample-quality judgments.

DDPM Figure 5 plots reconstruction RMSE on a [0, 255] scale against reverse steps and received bits per dimension. The distortion drops rapidly at low coding rates in this CIFAR10 experiment. This is a progressive reconstruction result, not a sampling-speed benchmark. A pleasing generated image and a short lossless code therefore remain separate evaluation targets. DDPM Figure 5 ↗.

A score differentiates log density

The gradient indicates the local direction of increasing log density. A normalization constant independent of x disappears under differentiation. A vector field can therefore be learned without explicitly evaluating the density everywhere.

For a Gaussian N(m,σ²I), the score is (m−x)/σ². It points toward the mean and increases in magnitude as the same displacement becomes less plausible under a smaller variance. A score describes a local density geometry; following it once is not a general sampling algorithm.

At positive Gaussian noise, discrete empirical clean samples become a smooth mixture. At zero noise, a density or its ordinary gradient may cease to exist. Endpoint limits require care.

Conditional targets recover a marginal score

A training pair supplies the conditional score. Squared-error regression averages that target over the clean posterior q(x₀|xₜ). The result is the marginal score required for generation. The conditional target and the learned marginal field are related, but they are not the same vector on every pair.

The posterior weighting is essential. Clean samples that explain the current observation contribute more. Averaging scores over the unconditional training-data distribution would generally yield the wrong field.

The score and denoising average encode the same conditional information at interior noise levels. That identity follows directly from the conditional Gaussian derivative.

Inspect ambiguity between two clean values

At x=0, the equal-prior clean values −2 and +2 are equally plausible. The posterior mean is zero. No clean example in this constructed dataset equals zero. A good squared-error denoiser can thus return a value outside the clean support.

Increase the observation toward +2. Its likelihood shifts posterior mass toward the positive clean value. Increasing σ weakens that evidence. The marginal score changes because both component directions and posterior weights change.

This is not proof that diffusion generation averages its output modes. It explains why a local denoising estimate and a final generated sample are different objects.

Change prediction coordinates, not the probability path

The coordinates describe the same noisy marginal. A model can predict clean data, noise, score, or a velocity-like linear combination. The sampler must convert the chosen output using the actual schedule.

At very low signal, dividing by aₜ amplifies error. At very low noise, dividing by σₜ amplifies error. A mathematically invertible interior transformation can still be numerically poor near an endpoint.

The symbol v is overloaded across papers. The linear combination above is not automatically the instantaneous velocity of any arbitrary flow-matching interpolation. Check the path definition before transferring a velocity target.

A model output is not automatically a score. The released VP and sub-VP wrappers negate the network output and divide by the corruption standard deviation. The VE wrapper instead treats the network output as the score. Time labels also differ between continuous and discrete models. Inspect the wrapper before applying another conversion, or the sign and scale can be applied twice. Released Score-SDE output wrappers ↗.

Equivalent outputs need not mean equivalent losses

Uniform score error therefore emphasizes a different set of times than uniform noise error. Uniform clean error changes the emphasis again. Equivalence of representations does not establish equivalence of finite-capacity training objectives.

Time sampling, explicit weights, output preconditioning, and coordinate reduction jointly determine the effective objective. An importance-sampled time distribution needs compensating weights if the intended integral is unchanged. Otherwise it intentionally changes the objective.

Check the total scale as well as relative weighting. A batch of large images and a batch of small latents can have different summed errors despite the same per-coordinate prediction quality. Losses and Objectives owns the general reduction rules.

The released continuous loss has two weighting branches. One multiplies the score residual by corruption standard deviation before squaring. The likelihood-weighted branch instead weights the squared score residual by diffusion amplitude squared. These are different training objectives, not just different names for the same sampler. The later likelihood-weighting reference is distinct from the original paper’s recommended weighting. Released Score-SDE losses ↗.

Pass from a chain to continuous dynamics

An SDE describes both drift and stochastic variation. A variance-preserving example uses $f(x,t)=-\beta(t)x/2$ and $g(t)=\sqrt{\beta(t)}$. The nonnegative rate beta controls signal attenuation and noise injection. A variance-exploding additive-noise process instead has zero drift and growing noise variance.

The time coordinate need not equal noise standard deviation or log signal-to-noise ratio. Reparameterizing time changes coefficients. A solver step in t cannot be replaced by the same number in σ without applying the chain rule.

The continuous formulation links discrete denoising chains and score-based generation. It also exposes different numerical sampling choices without changing the target field. Score-Based Generative Modeling through SDEs ↗ gives this framework.

Sub-VP keeps the VP signal attenuation but injects less noise. Define $B_t=\int_0^t\beta(u)\,du$ for a nonnegative rate. Both processes multiply the clean state by $e^{-B_t/2}$. Their clean-conditioned variances differ:

These variances condition on a fixed clean state; they are not the total data-population variance. Sub-VP uses diffusion amplitude $\sqrt{\beta(t)(1-e^{-2B_t})}$. Simply swapping a sampler while retaining VP corruption targets would not train this process. Score-SDE Appendix B ↗.

For integrated beta log(2) and clean value 2, both means are 1.4142. VP conditional variance is 0.50, while sub-VP variance is 0.25. This is an analytic example, not a quality comparison. For positive finite integrated beta, sub-VP has smaller conditional variance. Both variances approach one as integrated beta grows without bound.

The VE variance difference assumes a smooth schedule with its stated initial noise. The paper’s practical geometric schedule treats time zero separately and solves above a positive cutoff. Likewise, “variance preserving” means unit population covariance stays unit under VP; it does not mean every dataset variance is constant.

In the released implementation, marginal_prob returns a mean and a standard deviation. Square its second output before comparing with the variance table. For sub-VP, this output equals the VP variance, but its role is still a standard deviation. The practical VE implementation uses its geometric sigma directly as the corruption standard deviation. It does not subtract the initial sigma squared. Match that corruption convention when reproducing training. Released Score-SDE marginal definitions ↗.

Reverse time without reversing the noise sign

For a decreasing grid, write $\Delta t=t_{\mathrm{next}}-t<0$. The deterministic update multiplies the complete reverse drift by that negative value. The random increment scales with $\sqrt{|\Delta t|}$, not $\sqrt{\Delta t}$.

An implementation can instead use increasing reverse time $\tau=T-t$. Then the drift changes sign to $-f+g^2s$. Both conventions are valid; mixing their signs is not.

The displayed formula assumes state-independent isotropic diffusion. General matrix or state-dependent diffusion needs the appropriate covariance and additional terms. A formula copied from this setting is not a universal SDE reversal recipe.

Condition the score at the current noise level

Bayes’ rule gives $\nabla_x\log p_t(x\mid y)=\nabla_x\log p_t(x)+\nabla_x\log p_t(y\mid x)$. For class labels, train the classifier on corrupted labeled examples at each noise level. Differentiate its log probability with respect to the noisy input, then add that gradient to the unconditional score. A classifier trained only on clean images does not supply the required noisy likelihood. Score-SDE §5 and Appendix I.1 ↗.

Exact conditional reversal also needs the appropriate terminal conditional distribution. Starting from the usual unconditional prior is justified only when terminal noise sufficiently removes information about the condition. Multiplying the classifier gradient by a different strength changes the target; it is not unchanged Bayesian conditioning.

For inpainting, the paper draws the observed coordinates from their forward-corrupted distribution and evaluates the unconditional score on the combined state. Only the unknown coordinates are evolved. This is an approximation to clean-observation conditioning, not a proof of exact posterior sampling. Colorization applies an orthogonal channel transform, completes unknown channels, then transforms back. Orthogonality preserves isotropic Gaussian noise. Score-SDE Appendix I.2–I.4 ↗.

Same marginals, different sample paths

The factor one half is not optional. The ODE replaces stochastic spreading with a deterministic transport whose continuity equation gives the same density evolution. It does not replicate individual Brownian trajectories.

The result is distributional: the ensemble at each time has the same marginal law. It does not identify which original clean datum produced a given noisy training observation. Approximate neural scores and finite numerical steps can break exact equivalence.

Do not remove randomness from a reverse SDE update and call the result the probability-flow ODE. Its drift also changes. Likewise, a deterministic sampler is not necessarily a literal Euler discretization in the same time coordinate.

A trajectory endpoint is not a denoising mean

In the constructed Gaussian process, the exact score is $-\frac{x}{1+t}$. The probability-flow velocity is $\frac{x}{2(1+t)}$. Its trajectory satisfies $x(t)=z\sqrt{1+t}$, so its clean endpoint is z.

At t=4 and z=1, $x=\sqrt5\approx2.2361$. The trajectory endpoint is 1, while the Bayesian denoising mean is $\frac{x}{5}\approx0.4472$. These are both correct answers to different questions. The observed noisy state could have arisen from many clean samples.

A consistency map should preserve the trajectory endpoint as t changes. Training it only to reproduce the clean posterior mean defines a different function. This distinction is the basis of the later consistency section.

Read DDPM Figures 6 and 14 as clean-image predictions along a sampled reverse chain, not as the noisy states themselves. Coarse structure appears before fine detail in these examples. Figure 7 instead fixes an intermediate latent and runs several stochastic completions. Those completions can share structure without producing identical pixels. These are different views of generation, not proof that a posterior mean is a trajectory endpoint. DDPM Figures 6–7 and 14 ↗.

Density requires the flow divergence

A smooth, well-defined flow supports the instantaneous change-of-variables formula. Density evaluation requires integrating both the state and the divergence. A high-dimensional Jacobian trace may need a stochastic estimator.

The resulting value is the likelihood of the modeled flow, not proof that the learned field equals the data distribution. Numerical tolerance, trace estimation, preprocessing and discrete versus continuous data conventions still matter.

This capability is distinct from fast sample generation. A one-step student can generate quickly while lacking a tractable exact density. Compare sample quality and likelihood as separate measurements.

The Skilling–Hutchinson estimator avoids constructing the full Jacobian. Draw a zero-mean probe with identity covariance, such as independent signs. Compute a vector–Jacobian product, then contract it with the same probe. Its expectation is the divergence; a single draw is not generally exact. Score-SDE Appendix D.2 ↗.

For a synthetic two-dimensional linear field, let the Jacobian be $J=\begin{pmatrix}2&3\\0&0\end{pmatrix}$. Its trace is 2. For independent sign probes $z=(z_1,z_2)^\mathsf{T}$, the estimate $z^\mathsf{T}Jz=2+3z_1z_2$ is either −1 or 5. Averaging all four sign pairs gives 2. The estimator can fluctuate even when the field is known exactly.

The paper uses RK45 with absolute and relative tolerances of $10^{-5}$ for likelihood evaluation. It averages five test-set runs and stops at a positive time cutoff. Table 2 evaluates uniformly dequantized CIFAR-10; its starred discrete DDPM entries are ELBO bounds. These conventions matter when comparing bits per dimension. More probes reduce trace-estimation noise, but do not remove solver, cutoff or model error. Score-SDE Table 2 and Appendix D.2 ↗.

The released likelihood routine draws one Gaussian or sign probe per data point and keeps it fixed throughout the ODE solve. Resampling inside each derivative evaluation would give the adaptive solver a randomly changing right-hand side. It integrates from a positive cutoff toward the prior, then adds the integrated divergence to the terminal log density. Its bits-per-dimension conversion includes the inverse data-scaling correction. This correction depends on the data representation; it is not a universal offset. Released Score-SDE likelihood routine ↗.

Separate ancestral sampling and deterministic updates

Why can the denoiser stay unchanged? DDIM preserves the clean-conditioned noisy marginals while changing their joint coupling across levels. The usual noise-prediction surrogate sees those marginals. However, identical weighted-loss optima are justified for separately parameterized levels; a finite shared network can respond differently to loss weights. Deterministic sampling is a limiting construction, not a positive-variance ELBO identity. DDIM §3–4 ↗.

The deterministic update retains the current inferred noise direction while changing signal and noise amplitudes. It can skip levels. It is not the operation of setting reverse variance to zero in every ancestral DDPM formula.

Stochasticity is neither automatically harmful nor automatically better. Its effect depends on score accuracy, numerical discretization and the sampling objective. DDIM ↗ introduces the non-Markovian construction.

For a scalar example, take current cumulative alpha 0.25, target cumulative alpha 0.64, state 1 and predicted noise 0.5. DDIM predicts clean state 1.1340, then combines signal 0.9072 with retained noise 0.3000. The next state is 1.2072. The corresponding ancestral mean with no fresh draw is 1.0371: removing randomness alone does not produce DDIM. These are synthetic scalar calculations, not image-quality measurements.

In the released DDIM sampler, index −1 denotes the clean endpoint: a prepended zero beta makes its cumulative alpha equal one. The generalized branch does not clip the clean prediction; the separate DDPM branch does. Match the branch, not just the repository name. Released DDIM update code ↗.

Inspect the actual timestep sequence. The released uniform runner uses integer division to choose its stride. Requesting 60 steps from 1,000 levels produces 63 indices, ending at 992 rather than 999. Rounded quadratic indices can repeat. A new implementation should validate ordered indices, actual evaluation count and its starting prior; do not silently equate requested steps with executed steps. Released DDIM sequence selection ↗.

DDIM Figures 5 and 9 hold the initial latent fixed while changing the sampling schedule. Similar high-level structure is an observed property of these examples, not exact equality between schedules. Figure 4 reports hours for 50,000 images on one Nvidia 2080 Ti; that timing is not a universal speedup. DDIM Figures 4–5 and 9 ↗.

Predict time, then correct at the new level

In the paper’s predictor–corrector procedure, the predictor advances to the next noise level. The corrector then runs score-based MCMC at that fixed level. Each corrector iteration evaluates the score again after changing the state. This is not Heun: Heun combines slopes to integrate a trajectory.

function langevinStep(x,score,step,z){
  if([x,score,step,z].some(v=>!Number.isFinite(v))||step<=0)
    throw new RangeError('Finite values and positive step required');
  return x+step*score+Math.sqrt(2*step)*z;
}

This tested scalar kernel uses a fixed positive step, the current score and an independent standard-normal draw z. It does not advance diffusion time. The paper also describes norm-based step selection; this kernel does not implement that adaptive rule. Score-SDE §4.2 and Appendix G ↗.

A finite Langevin step is approximate even with an exact score. For a standard-normal target, score equals minus the state. Starting at variance one, step 0.1 gives variance 1.01 after one update. This synthetic moment calculation shows discretization bias, not a neural-model result. More correction costs more score evaluations; compare samplers at equal evaluation budgets.

In Score-SDE Figure 9, rows change total score evaluations; columns change corrector steps per level. Compare within a row to inspect budget allocation. Table 1’s 2,000-step predictor baseline also requires interpolation of a model trained at 1,000 discrete levels. The paper tunes corrector strength and adds final denoising. Its favorable PC results are not a guarantee that correction always helps. Score-SDE Figure 9 and Appendix G ↗.

The released PC loop applies the corrector at the current time, then applies the predictor. With final noise removal enabled, it returns the last predictor mean rather than its random sample. Its adaptive Langevin step uses score and noise norms averaged across samples and devices; this is not independent per-example normalization. Keep operation order, norm reduction and final noise removal explicit when reproducing a sampler. Released Score-SDE sampler ↗.

See numerical error in a known field

Euler uses the current slope once. Heun predicts a provisional endpoint, evaluates another slope, then averages the two slopes. In this toy field, Heun reduces error per step but spends twice as many field evaluations.

Compare at equal evaluations as well as equal step counts. Four Heun steps and four Euler steps do not have the same field-call budget. Higher formal order does not eliminate a wrong learned score, an incompatible schedule or an unstable step size.

The exact exponential is available only because the demonstration uses a known scalar field. No plotted line represents FID, prompt fidelity, neural training or measured serving latency.

Identify which error more steps can reduce

Local reverse predictions can be easier to learn than a global noise-to-data map. However, a larger number of training levels does not universally make every target easier or improve sample quality monotonically. Noise schedule, model capacity and objective determine that behavior.

Inference cost depends on number of field evaluations, model cost, guidance calls, batching, memory traffic and the decoder. A nominal one-step sampler may still run an encoder or decoder and more than one network branch. Count the complete workflow.

Compress a chosen noise-to-sample coupling

This approach learns a particular coupling between input noise and teacher outputs. Generating targets can be expensive, whether stored offline or recomputed during training. The student must approximate a much larger transport operation than one local denoising step.

A deterministic teacher assigns one target to each exact input. Squared error does not inherently average different teacher modes for that same input. Averaging occurs when the target remains random conditional on the student input, or when the student lacks information or capacity.

If teacher randomness matters, include its seed/noise in the student input or choose an objective that matches a distribution. A reproducible target convention is part of the algorithm, not merely a dataset detail.

Replace two teacher steps with one student step

Progressive distillation starts from a deterministic teacher. It builds a target whose single student update reaches the same state as two teacher updates. The trained student becomes the next teacher, and the schedule is halved again.

For a power-of-two schedule, 64→32→16→8→4→2→1 takes six reductions. Each stage trains a model; six reductions do not mean six minibatches. Non-power-of-two schedules require an explicit partition and endpoint convention.

The method preserves local teacher behavior before demanding a full one-step map. Teacher errors and student approximation errors can propagate across stages. Progressive Distillation ↗ supplies the staged procedure.

One student target, two teacher evaluations

Consider a synthetic scalar example with cumulative alpha squared 0.25 → 0.50 → 0.81. Start from state 1. The first teacher predicts clean value 2; the second predicts clean value 1 at the new state. Those predictions need not agree.

The effective student prediction is neither teacher prediction. Substituting that target into one DDIM update reproduces the teacher endpoint. A trained student only approximates this target. The numbers are computed from the scalar kernels; they are not image-quality measurements.

Construct the training pair before changing the schedule

The original procedure still samples training data and corrupts it on the student’s discrete time grid. It does not require full teacher rollouts, but it is not data-free. Initialize the student from the teacher. Evaluate the frozen teacher twice, with the second evaluation at the first predicted state. Stop gradients through the target. After training this stage, promote the student to teacher.

Each student specializes to its trained grid. Arbitrarily changing inference times is not the same as evaluating the original continuous denoiser. The paper keeps teacher and student architectures the same; its main speed gain is fewer evaluations, not fewer parameters.

Keep the pure-noise prediction well defined

The paper includes an exactly zero-signal terminal level. Ordinary noise-to-clean conversion divides by signal amplitude and becomes singular there. Direct clean prediction or the velocity coordinates defined earlier avoid that division. Noise-error weighting also gives zero clean-reconstruction weight at zero signal-to-noise ratio (SNR). The paper instead examines weights $\max(\mathrm{SNR},1)$ and $1+\mathrm{SNR}$. Both retain nonzero weight at pure noise.

These choices do not make every output and weighting combination stable. The undistilled CIFAR-10 ablation reports divergence for noise output with truncated-SNR weighting. Treat parameterization, weighting and time sampling as a coupled design. Progressive Distillation §3–5 and Appendix G ↗.

The main comparisons retain useful quality at four or eight sampling steps, with more degradation at one or two. The stochastic baseline is tuned separately for each step budget. CIFAR-10 curves average four seeds; the larger datasets use one run. These conditions do not establish universal quality preservation or a universal training-cost saving. The original schedule trains 50,000 updates per stage, and 100,000 for the final two- and one-step stages; faster schedules trade training cost against quality. Progressive Distillation §5.2 and Appendices E–I ↗.

Match the released endpoint convention

The released JAX code uses finite cosine log-SNR bounds, rather than exact zero signal and zero noise. Training samples an integer index uniformly on the student grid. The teacher midpoint is halfway in schedule time, not halfway in cumulative alpha squared.

Both teacher calls and the student training prediction are unclipped. The lowest-index target is explicitly replaced by the second teacher’s clean prediction. The sampler likewise returns the clean prediction on its last step. Applying the ordinary finite-bound DDIM formula there gives a slightly different endpoint. Released target and sampler ↗; Released schedule ↗.

The released CIFAR configuration predicts clean data with truncated-SNR loss. Its log-SNR bounds are −20 and 20. Evaluation clips the clean estimate; training does not. Distillation disables dropout and parameter averaging in this configuration. These settings are reproduction details, not requirements for every distillation method. Released CIFAR distillation configuration ↗.

Learn the common endpoint of a trajectory

A consistency function maps every point on a trajectory to its common near-clean endpoint. The endpoint belongs to the ODE coupling. It is not generally the original datum used to create a noisy training pair.

Self-consistency alone allows a useless constant function. The boundary condition prevents that solution by requiring identity behavior at the minimum time. A skip-connected parameterization can enforce this exactly.

One-step generation samples the appropriate terminal prior and applies Fθ once. Few-step consistency sampling can revisit intermediate levels. More evaluations may improve fidelity, but the result depends on the trained model and sampling procedure.

Enforce the positive-time boundary

Appendix C shifts the skip and output numerators by the positive cutoff. The output denominator still contains t, not t−ε.

At the cutoff, the skip is one and the network coefficient is zero. The output is therefore the input, regardless of network weights. Consistency Models Appendix C ↗.

Distillation and training construct different targets

Consistency distillation uses a pretrained denoising teacher to connect neighboring states. The loss compares endpoint predictions, not the states themselves. The target branch is stopped while the online model updates.

Consistency training instead constructs learning targets from data without a pretrained diffusion teacher. Its estimator and discretization analysis are method-specific. It is not exact teacher integration performed without a teacher, nor generic independent noise-pair regression.

Noise levels, distance function, target-network update and boundary coefficients jointly define training. Consistency Models ↗ distinguishes training and distillation. Neither consistency equality nor an EMA update alone establishes distributional fidelity.

Reuse the noise, not a claimed ODE trajectory

For teacher-free consistency training, sample a datum and one Gaussian noise vector. Use that same vector at neighboring noise levels. In this section, time equals noise standard deviation; it is not the VP cumulative alpha coordinate.

The conditional score estimate is negative noise divided by its standard deviation. Substituting this estimate into one backward Euler update gives the shared-noise lower state. The score identity holds after conditioning and averaging. It does not identify each training pair with an exact ODE path.

This synthetic example uses standard-normal training data. The high state is 3. The three lower states differ even with an exact score. The shared-noise pair is a training construction, not an image-generation experiment.

The original asymptotic argument assumes an exact marginal score, Euler updates, smooth target and distance functions, bounded derivatives and weighting, and finite score moments. Taylor expansion makes the expected first-order terms agree as the grid is refined. Unbiased scores do not make nonlinear losses exactly equal at finite spacing. Consistency Models §5 and Appendix A.3 ↗.

Multistep inference is different: predict an endpoint, inject fresh noise to a lower chosen level, then predict again. It does not apply the endpoint map repeatedly to an unchanged noisy state.

Read the consistency experiments with their settings

The main experiments initialize consistency distillation (CD) from pretrained EDM diffusion weights and consistency training (CT) randomly. Both use the learned perceptual similarity metric LPIPS and Rectified Adam, without warmup, learning-rate decay or weight decay. LPIPS receives bilinear 224×224 inputs on CIFAR-10/ImageNet, not LSUN.

Seed-aligned sample grids illustrate coupling, not absence of mode collapse. Editing panels use an LSUN Bedroom CD model with inference-time constraints, not task-specific training. Consistency Models §6, Table 1 and Appendices C–E ↗.

A continuous pseudo-objective is not a loss metric

Appendix B’s stopped-target continuous objectives are pseudo-objectives: their gradients define training, but their values do not measure convergence. Its CT comparison initializes both variants from EDM; it is not the random-initialized main experiment. Consistency Models Appendix B ↗.

Connect the consistency equations to released code

The released PyTorch implementation covers ImageNet-64 and LSUN; CIFAR-10 uses a separate JAX repository. Its flag named distillation selects boundary coefficients for both consistency training and distillation. Released repository scope ↗.

The raw U-Net output is not the endpoint prediction. Scale the noisy input before the network; then combine the network output with the skip path.

The released CT launch examples supply teacher_model_path. The entry point copies those weights into the online model, but the CT loss branch does not receive the diffusion teacher. Teacher-free targets do not imply random initialization. Pinned training entry point ↗; Pinned training loop ↗.

Count evaluations, not schedule entries

Each pair predicts an endpoint, then adds fresh noise for the next selected level. The final marker ideally adds zero noise; floating-point schedule arithmetic can leave a tiny residual. Three indices require two network calls. Sampling clips the denoised output by default; training does not.

This scalar reference uses explicit strictly decreasing levels within [0.002,80]. The last entry defines the exact cutoff. Callbacks must return finite scalars. It is not the released tensor implementation or a trained generator:

from math import sqrt
def cm_sample(endpoint, x, levels, indices, normal):
    if len(levels)<2 or any(not 0.002<=s<=80 for s in levels): raise ValueError("levels")
    if any(b>=a for a,b in zip(levels, levels[1:])): raise ValueError("decreasing levels")
    if any(type(i) is not int for i in indices): raise ValueError("integer indices")
    if len(indices)<2 or indices[0]!=0 or indices[-1]!=len(levels)-1: raise ValueError("endpoints")
    if any(b<=a for a,b in zip(indices, indices[1:])): raise ValueError("increasing indices")
    cutoff = levels[-1]
    for a,b in zip(indices, indices[1:]):
        x = endpoint(x, levels[a])
        x += sqrt(levels[b]**2-cutoff**2)*normal()
    return x

Pinned loss, wrapper and sampling implementation ↗.

Verify the recipe wiring before a run

In this pinned revision, the training entry point does not forward loss_norm to the denoiser constructor. Selecting an L2 CLI flag alone therefore retains default LPIPS. The loop condition remains true when lr_anneal_steps is zero, even after total_training_steps. Check both paths before copying a launch command. Pinned constructor and defaults ↗; Pinned launch examples ↗.

Derive velocity from the declared interpolation

A probability-flow ODE comes from the diffusion field. Flow matching can instead start from a chosen interpolation and regress its velocity directly. Define the endpoints before using any velocity formula:

A negative time increment moves this path toward the clean value. Reversing the time coordinate reverses the velocity sign. Reusing the same sign under both conventions changes the sampler.

The marginal velocity averages over pairs consistent with the observed state. It is not generally the displacement to one original clean sample. FGM v1 equation 3.5 is inconsistent with its equation 3.6 interpolation; the derivative above follows the declared path. Flow Generator Matching §3 ↗.

Learn displacements at multiple step sizes

A shortcut model predicts a displacement per unit step at a declared time and step length. Its output can support local movement or a long jump. The step-length input changes the learned function, not just the number of times it is called.

The shortcut base target comes from flow matching at infinitesimal step size. Longer targets bootstrap two smaller learned jumps, with stopped target gradients. The second prediction uses the first predicted state and advanced time, not the straight dataset interpolation.

For the toy ODE $x^{\prime}=-x$, an exact finite-step displacement is $\frac{x(e^{-d}-1)}d$. Averaging the two consecutive displacements reproduces the 2d displacement exactly. A learned network only approximates this composition.

Joint training avoids separate distillation stages but still incurs bootstrap evaluations and approximation error. One Step Diffusion via Shortcut Models ↗ defines this distinct approach.

Build a shortcut target without changing its meaning

The base loss grounds the network in observed noise–data pairs. The bootstrap loss then teaches a longer displacement from two shorter predictions. Use the same noise-to-data convention throughout:

The recipe uses 75% empirical examples and 25% bootstrap examples. Its eight positive step lengths run from 1/128 to 1 by doubling. For the smallest bootstrap half-step, query the field with q=0; otherwise q=d. The integration distance remains d. Choose grid-aligned times that leave room for both halves.

function shortcutTarget(field,x,t,d){
  finite(x,t,d);
  const level=Math.log2(128*d);
  if(typeof field!=='function'||!Number.isInteger(level)||level<0||level>6)
    throw new RangeError('Dyadic half-step from 1/128 to 1/2 required');
  if(t<0||t+2*d>1)throw new RangeError('Target extends outside [0,1]');
  const query=d===1/128?0:d;
  const first=field(x,t,query);finite(first);
  const mid=x+d*first;finite(mid);
  const second=field(mid,t+d,query);finite(second);
  const target=first/2+second/2;finite(target);
  return {first,mid,second,target,studentStep:2*d};
}

This scalar reference constructs target values only. A tensor implementation must freeze the EMA branch and stop target gradients explicitly. Its callbacks do not implement automatic differentiation, guidance, batching or the training-time sampler.

This is our arithmetic example, not a learned flow or image result. Keeping the old time in the second call would instead give 2.390625. The discrepancy isolates why advancing time matters; the predicted midpoint also differs from a new empirical interpolation.

The reported recipe uses EMA decay 0.999, AdamW learning rate 0.0001 and weight decay 0.1. Guidance applies only to zero-step field queries; its scale is chosen before training. Do not apply guidance again to a learned long jump. Shortcut Models v3 §3.1 and Appendix B ↗.

The two target calls affect only the bootstrap quarter. Assuming a backward pass costs two forward passes, an update costs 3.5 forward-equivalent units instead of 3: about 16.7% extra. This is a compute estimate, not a measured runtime guarantee.

Read shortcut quality against its sampling budget

At inference, start from noise at time zero. For N equal steps, call the model with length 1/N, advance the state by that length times its output, then advance time. N=1 needs one call. The trained dyadic budgets do not certify arbitrary untrained step lengths.

An asterisk marks a budget the objective was not intended to support. The comparison uses the same architecture and codebase, but baselines can receive greater compute. Shortcut preserves flexible sampling budgets; progressive distillation has the better one-step FID here. One-step shortcut quality does not match its own 128-step quality.

The ImageNet recipe uses batch size 256, 800k updates and guidance scale 1.5. Images are encoded as 32×32×4 latents. Evaluation uses EMA weights and 50k samples against full-dataset statistics. Shortcut Models v3 Table 1 and Appendix B ↗.

The sample grids reuse initial noise within each column. They show approximate preservation of image content across budgets, not pixel identity. Reducing 128 calls to one is not evidence of 128× end-to-end speedup: decoding, launch overhead and hardware still matter.

Match distributions without teacher rollouts

Paired distillation preserves a teacher transport. Distribution matching asks a different question: does the student generate the desired population, even if individual noise seeds map differently? A score-based objective can compare smoothed student and target distributions.

Positive noise makes the student distribution easier to describe with a score. The generator changes both the sampled states and its own marginal score. Differentiating the displayed norm while ignoring either dependence changes the intended gradient.

An auxiliary denoiser can learn the student score from generated examples. Its predictions are estimates, not an analytic score of the generator. Training quality depends on that estimate and its update speed as the generator changes.

Avoiding teacher rollout datasets does not eliminate teacher calls, auxiliary training, or backward passes through input-dependent fields. The saved cost and the new cost must both be measured.

SiD requires a gradient contract, not just a norm

SiD alternates a generator update with an auxiliary denoiser fitted to corrupted student samples. Its teacher is fixed. The generator loss uses score identities and a fused objective rather than only the score-difference norm.

The original method retains derivatives of both denoisers with respect to xₜ during the generator update. Freezing their parameters is not detaching their outputs. The auxiliary update uses generated-data denoising targets; the generator update uses the fused loss.

Weights, noise sampling and gradient routing are essential. An arbitrary stop-gradient placement produces a different algorithm. The identity objective does not imply every variant uses a GAN discriminator. Score identity Distillation, §4 and Algorithm 1 ↗ is the implementation reference.

Why equal score losses can give different gradients

Why is a good auxiliary estimate not enough? Consider scalar Gaussian data with mean zero and unit variance. The generator shifts that mean by θ. Its auxiliary denoiser has a separate mean parameter ψ, held fixed during the generator update.

The denoiser difference is constant in its input. Its squared norm therefore gives no generator gradient. Setting ψ numerically equal to θ does not connect their computation graphs. The fused cross term restores a dependence on the generated sample.

Equal values can hide different gradients; a zero surrogate value can still provide an update. Here the positive derivative moves the generator mean toward zero under gradient descent. Its scale need not match the ideal discrepancy.

Freeze denoiser weights, but retain both denoisers’ input derivatives. Do not infer that λ is irrelevant in neural training from this affine example. The paper uses a different, detached adaptive weight; the constant weight here isolates the gradient mechanism. SiD Appendix D ↗ supplies the Gaussian setup; the table evaluates equation 23 with the stated constant weight.

Keep the two SiD update distributions separate

Both trainable networks start from pretrained EDM weights. Data-free distillation does not remove pretraining or evaluation data. Update the auxiliary denoiser first, with generated targets and EDM’s denoising sampler and weighting:

Handle zero or nonfinite denominators explicitly; the displayed expression supplies no safeguard. Do not silently introduce epsilon clipping. The released denominator floor is explicit in the next section.

function sidGeneratorNoise(t){
  if(!Number.isFinite(t)||t<0||t>.8)
    throw new RangeError('Generator time outside [0,.8]');
  const low=.002**(1/7),high=80**(1/7);
  return (low+t*(high-low))**7;
}

SiD Algorithm 1 ↗; equations 25–26 ↗.

Translate SiD’s loss into the released update

The pinned main-branch implementation is SiD, not the separate adversarial SiDA branch. It cancels the explicit fourth power of noise through the chosen weight. The denominator becomes a stopped mean absolute residual across channels and spatial coordinates:

The loop sums coordinate losses, then scales by the per-rank total batch size and the configured loss scale. Replacing this sum with a pixel mean changes gradient magnitude. Microbatches accumulate before each optimizer step.

This finite-scalar adaptation checks loss values only. It does not implement autograd, float32 rounding or the release’s NaN-sample filtering:

function sidCoordinateLoss(y,teacher,auxiliary,mixing=1.2){
  if(!Number.isFinite(mixing)||![y,teacher,auxiliary].every(a=>
    Array.isArray(a)&&a.length>0&&a.length===y.length&&Array.from(a).every(Number.isFinite)))
    throw new RangeError('Aligned finite coordinate arrays required');
  const scale=Math.max(1e-5,y.reduce((s,v,i)=>s+Math.abs(v-teacher[i]),0)/y.length);
  const loss=y.reduce((s,v,i)=>{
    const delta=teacher[i]-auxiliary[i];
    return s+delta*((1-mixing)*delta+auxiliary[i]-v)/scale;
  },0);
  if(!Number.isFinite(scale)||!Number.isFinite(loss))throw new RangeError('Nonfinite loss');
  return {scale,loss};
}

The small floor prevents division by zero; it does not bound the loss or gradient. Rejecting nonfinite inputs here is stricter than dropping NaN samples. Audit the all-filtered batch case before treating that release path as robust.

The runner still reads a dataset for shapes, class labels, sample grids and evaluation. Image pixels are overwritten by generated samples before the loss. Data-free objectives therefore do not imply a dataset-independent launch script.

The FFHQ launch uses generator Adam β₁=0.9, agreeing with Table 6 rather than Appendix A’s value 1. Pinned loss ↗; update loop ↗; launch recipes ↗.

Read SiD quality and stability with their settings

FID uses 50k images and the teacher-training reference set. Evaluate EMA checkpoints every 500k synthetic images and select the lowest FID. Ten re-evaluations are not ten independent training runs.

Training uses 500M synthetic images, extended to 1B for ImageNet with λ=1.2. ImageNet’s batch increase from 1024 to 8192 also lowered the learning rate from 5e−6 to 4e−6. FP16 is a suspected cause, not an established diagnosis. Figures 4/9 show spikes. Log–log linearity indicates power-law behavior, not exponential convergence. Table 6 excludes FID-evaluation cost. SiD Tables 1/3 and Figures 4/9 ↗; Appendices A/E ↗.

Flow Generator Matching retains the missing gradient

Flow Generator Matching (FGM) extends the distribution-matching viewpoint to induced flow fields. A generator’s noised/interpolated population has an implicit marginal velocity. An auxiliary model estimates that field from conditional flow targets.

The second term accounts for a dependence that naive frozen-field matching omits. The theoretical full surrogate and the practical recipe are not identical: the paper drops L₁ in its CIFAR-10 and text-to-image experiments because of instability.

FGM is not a claim that the one-step generator executes a teacher ODE. It trains a population through vector-field identities. Flow Generator Matching, §4–5 ↗ explains the derivation and practical modification.

See which generator gradient field matching misses

Reuse the mean-shift idea from the SiD example, but now use the linear flow path. Real clean data has mean zero and variance one. Generated clean data has mean θ and variance one. An exact auxiliary field at the current generator mean still needs the correct derivative contract.

Both marginal fields have the same input slope. Their difference is constant, so moving generated inputs gives no gradient through the naive norm. The conditional velocity still depends on the generated clean sample. FGM’s product identity uses that dependence to recover the missing term.

At an exact auxiliary fit, the correction recovers the ideal derivative in this example. This does not prove that dropping the first term is correct for arbitrary neural fields. Approximate auxiliary fits and modified training losses remain separate assumptions. FGM Appendix A ↗.

Separate FGM theory from the image-training recipe

Table 4 reports 20M training images, batch 512 and Adam at 2e−5. The CIFAR distillation recipe uses logit-normal times, not Algorithm 1’s uniform-time example.

The SD3-Medium recipe adds GAN training on roughly 500k synthetic images and omits the first FGM term. Its result is not evidence for the unmodified data-free theoretical objective. Generator and auxiliary losses use different timestep distributions.

GenEval reports 0.65 for the one-step model versus 0.70 for 28-step SD3. Evaluation uses four images per prompt at 1024px and model-specific guidance. Fewer steps are not a measured end-to-end latency ratio. FGM Tables 1–2 ↗; Table 3 ↗; Appendix B ↗.

The broader corruption view has different mathematics

The broad design pattern is to define corruption and learn recovery. Gaussian noise is one choice. Masking, token replacement, deletion, blurring and partial observation expose different missing-information tasks.

The connection is conceptual until a process and an objective are specified. A masked language-model predictor is not automatically a normalized iterative generator. A blur may erase information without becoming a simple known terminal prior.

Discrete token probabilities do not have the ordinary Euclidean gradient used by a continuous Gaussian score. They need discrete transition kernels, reverse conditional probabilities or another declared probability path. Autoregressive generation can also be viewed through ordered partial observation, but its factorization is different.

A diffusion drafter for language-model speculation belongs to Speculative Decoding. That note owns verification and token-level acceleration. The Gaussian image equations here must not be transferred to such a drafter without a discrete-process derivation.

Choose the change that addresses the bottleneck

There is no universal ordering from low to high training complexity. A large frozen teacher plus an auxiliary model can dominate memory. A costly offline teacher dataset can dominate paired training. A solver change can be cheap to develop yet fail at the desired evaluation budget.

Compare quality at matched conditioning, resolutions, data and evaluation settings. Include diversity, coverage, prompt alignment and artifact rates, not only visual sharpness. A teacher checkpoint and its sampling configuration together define the target.

The Score-SDE comparisons illustrate why settings matter. Table 3 selects the lowest-FID checkpoint and uses predictor–corrector sampling. Table 2 uses the final checkpoint and black-box ODE sampling. Do not attribute every difference to the SDE alone. Appendix H’s architecture exploration also uses different checkpoint windows for VE and VP models. Score-SDE §4.4 and Appendix H ↗.

Small kernels and necessary checks

The first useful kernel is a direct training sample. The schedule lookup must return batch-shaped coefficients with the correct signal/noise convention. The following is pseudocode, not a complete training loop.

t = sample_noise_level(batch_size)
a, sigma = schedule(t)  # reshape for coordinate broadcasting
eps = randn_like(clean)
noisy = a * clean + sigma * eps
prediction = denoiser(noisy, t, condition)
per_example = squared_error(prediction, eps).mean(data_axes)
loss = (time_weight(t) * per_example).mean()

For additive-noise score checks, test the analytic posterior identity before adding a neural field. For a DDPM sampler, test the posterior coefficients, noise conversion, schedule indexing and final-step convention on a known Gaussian population.

Make the ancestral last step explicit

This tested scalar JavaScript kernel accepts cumulative alpha at the previous level, the current beta, state x, predicted noise eps and standard-normal draw z. The kernel rejects nonfinite inputs. A tensor implementation must broadcast schedule values and preserve data shapes.

function ancestralStep(abarPrev,beta,x,eps,z,mode='posterior'){
  if([abarPrev,beta,x,eps,z].some(v=>!Number.isFinite(v)))
    throw new RangeError('Finite values required');
  const invalidSchedule=abarPrev<=0||abarPrev>1||beta<=0||beta>=1;
  const invalidMode=!['posterior','beta'].includes(mode);
  if(invalidSchedule||invalidMode)throw new RangeError('Invalid DDPM step');
  const alpha=1-beta,abar=abarPrev*alpha;
  const mean=(x-beta*eps/Math.sqrt(1-abar))/Math.sqrt(alpha);
  const variance=mode==='beta'?beta:beta*(1-abarPrev)/(1-abar);
  const noiseStd=abarPrev===1?0:Math.sqrt(variance);
  return {mean,variance,noiseStd,next:mean+noiseStd*z};
}

Previous cumulative alpha equals one at t=1. The kernel then returns the mean without fresh noise, including when mode selects beta variance. The two fixed variance choices are not the forward marginal variance. A likelihood calculation still needs its separate decoder convention. DDPM §3.2–3.3 and Algorithm 2 ↗.

This kernel implements the unclipped equation, not the complete released image sampler. The original TensorFlow code first converts predicted noise into a clean-image estimate. By default, it clips that estimate to [−1, 1], then computes the posterior mean. Clipping can change the update; it depends on the data range and is not a universal rule for latent states.

The released implementation uses posterior variance and zero-based schedule indices. Code index 0 means one forward corruption step; the final reverse update therefore masks fresh noise at index 0. The paper calls that step t=1. Schedule coefficients are calculated in float64, then stored as TensorFlow constants in the selected dtype. Released DDPM sampling implementation ↗.

dt = next_time - time  # negative in this convention
score = score_model(x, time)
drift = forward_drift(x, time) - diffusion(time)**2 * score
x = x + drift * dt + diffusion(time) * sqrt(-dt) * randn_like(x)

This is a schematic Euler–Maruyama reverse-SDE update. It assumes scalar state-independent diffusion and dt<0. Removing the random term does not make it the probability-flow update; halve the score coefficient for that ODE.

Distillation tests must include gradient ownership. Confirm which parameters update and whether teacher/auxiliary input derivatives remain live. Check schedule endpoints, nonfinite values, tiny denominators and loss-weight scales. First validate a low-dimensional known distribution; then evaluate real-data generation.

A chronology with parallel acceleration branches

These branches coexist. A newer generator objective does not replace numerical solvers for every task. Diffusion probabilistic models and score matching also predate DDPM; the table tracks this note’s method sequence, not the full history of generative modeling.

The practical progression is local prediction → reliable transport → fewer evaluations. Each acceleration method chooses a different object to preserve: field, trajectory endpoint, composed displacement, or output population.

Sources and connected topics

ddpm ↗ · Score SDE ↗ · ddim ↗ · progressive ↗ · consistency ↗ · Score identity Distillation ↗ · shortcut ↗ · Flow Generator Matching ↗

Su Jianlin’s original series references: Article 9119 ↗ · Article 9152 ↗ · Article 9209 ↗ · Article 9262 ↗ · Article 9509 ↗ · Article 10085 ↗ · Article 10567 ↗ · Article 10617 ↗ · Article 10633 ↗

Distillation · Losses and Objectives · Probability · Calculus · Information Theory · Speculative Decoding

All four interactive examples use exact synthetic distributions or known scalar dynamics. None runs a trained image generator or displays measured generation quality.