Losses and Objectives

What the objective asks for

These are population statements. A finite dataset, restricted model family and imperfect optimizer can produce a different result. A low training loss also does not establish good calibration or out-of-distribution performance.

Magnitude, direction and the unit sphere

Before using a norm in an objective, separate magnitude from direction. The Euclidean norm is distance from the origin: $\|x\|_2=\sqrt{x^\top x}$. For x = [3,4], the norm is 5, the squared norm is 25, and the normalized vector is [0.6,0.8].

An orthogonal transformation R satisfies $R^\top R=I$. Therefore $\|Rx\|_2^2=x^\top R^\top Rx=\|x\|_2^2$. A rotation or reflection preserves Euclidean length. This symmetry is why spherical geometry fits isotropic Gaussian models. It does not imply that learned embeddings are isotropic.

The zero vector has no direction. Numerical implementations need a policy. PyTorch uses $\frac{x}{\max(\|x\|_2,\varepsilon)}$: zero stays zero, and vectors with norm below ε are not unit length. Choose the embedding axis explicitly. PyTorch normalization contract ↗.

Unit length alone cannot prevent collapse. If every example maps to the same unit vector, the representation is still collapsed. Whether a self-supervised method normalizes embeddings and how it prevents collapse are separate questions. Self-Supervised Vision Learning →

The next section applies norms to residuals. Later, parameter penalties apply the same geometry to weights. Vectors and Geometry →

L1, L2 and regression

For a vector x, the L2 norm is $\|x\|_2=\sqrt{\sum_i x_i^2}$. Its squared norm is $\|x\|_2^2=\sum_i x_i^2$. For residuals $r_i=\hat y_i-y_i$, mean squared error divides the squared sum by the number of included elements.

Squared error penalizes a large residual quadratically. Its derivative is proportional to that residual. Absolute error penalizes it linearly, with a bounded subgradient away from zero. Nondifferentiability at zero does not make L1 optimization inherently unstable.

A Gaussian observation model with fixed variance gives a negative log-likelihood proportional to squared error. A Laplace observation model with fixed scale gives absolute error. If the model learns the Gaussian variance, its log-likelihood must also include the log-variance term; it cannot simply inflate variance to erase residuals.

Cross-entropy from logits

Let p be the target distribution and z the model logits. Softmax gives $q_i=\frac{e^{z_i}}{\sum_j e^{z_j}}$. Cross-entropy evaluates the predicted probabilities under the target.

For a hard label y, p is one-hot and $L=-\log q_y$. In a dataset, that observed label is a sample from a conditional distribution. The expected score over repeated observations is minimized by the underlying distribution, not necessarily a one-hot prediction.

Cross-entropy equals $H(p)+D_{\mathrm{KL}}(p\|q)$. The first term is fixed when the target is fixed. This is why forward teacher-to-student KL and teacher cross-entropy have identical student gradients at the same temperature. Reverse KL is a different objective. See Information Theory and Compression → for the coding derivation.

Use a fused logits-based loss. Computing softmax, then taking log of a rounded probability, can underflow. Finite forward values still do not guarantee accurate gradients; the next section separates those requirements. Label smoothing changes p itself. Temperature changes q and gradient scaling. Treat both as objective choices, not harmless numerical fixes.

Avoiding overflow is not enough

Subtracting the largest logit prevents exponential overflow. It does not protect every small quantity used by the backward pass. A model can have finite logits and loss while receiving an inaccurate gradient. Barron, Gnanapragasam and Tschannen (2026) examine this failure in A Fix for Softmax and Cross-Entropy ↗.

Why a correct confident prediction can lose its gradient

Consider a hard target t that already has the largest logit. Combine all other classes into one residual mass r. Define the margin against their log-sum-exp, not only against the runner-up:

The exact target probability is $q_t=1/(1+r)$. When r is small enough, floating-point addition rounds 1+r to 1. A standard backward path can then evaluate the target gradient as 1−1=0. The other gradients still have small positive numerators. They need not vanish at the same point.

The paper calls this softmax collapse. Its approximate round-to-nearest cliffs are m≈5.5 in BF16, 16.6 in FP32 and 36.7 in FP64. Actual kernels may upcast intermediate operations; record that arithmetic contract instead of inferring it from model-weight dtype.

Rewrite the computation, not the objective

Keep the hard-target term out of the exponential sum and apply a stable softplus to the log-sum of the remaining terms. This is still ordinary cross-entropy. Unlike replacing softmax with StableMax, it does not introduce a different prediction map.

For the paper's FP32 example z=[9,−9,−9], t=0, its tested standard JAX/Optax path returns zero loss and zero target gradient. The rewritten path returns loss ≈3.046×10⁻⁸ and gradient ≈[−3.046×10⁻⁸,1.523×10⁻⁸,1.523×10⁻⁸]. These are reported implementation results, distinct from the scalar demonstration above.

# PyTorch sketch; z: [..., classes], t: [...] valid integer IDs.
# Require finite logits and at least two classes. No reduction here.
def hard_target_ce(z, t):
    z_t = z.gather(-1, t[..., None])
    target = F.one_hot(t, z.shape[-1]).bool()
    others = (z - z_t).masked_fill(target, -torch.inf)
    return F.softplus(torch.logsumexp(others, dim=-1))

For finite normal logits, useful checks include a high-precision reference, common-shift invariance and a near-zero sum of logit gradients. Inspect the target gradient separately. A uniform zero-sum projection is not the same as reconstructing the missing target derivative; it redistributes the error across classes.

The evidence concerns small modular-arithmetic grokking tasks and controlled distillation experiments, not a demonstrated large-LLM speedup. The paper's FP32 modular-addition comparison reports 94.3±5.4% versus 100±0% test accuracy across five seeds, with AdamW, weight decay 1 and ε=10⁻⁸. Do not generalize that gain to another training recipe.

A different error appears when teacher and student already agree: see distillation near an exact match. For the optimizer consequence, see Slingshot and precision cliffs.

Hard labels, empirical averages and teachers

A hard label is one observation, not the complete conditional distribution. Fix an input x, draw y from p(·|x), and average its loss:

For eight illustrative observations with class counts [3,2,3], the empirical target is [3/8,2/8,3/8]. Averaging eight one-hot log losses equals cross-entropy against that empirical distribution. It need not equal the population risk.

A teacher supplies a soft target directly. Its probabilities express relative plausibility beyond the observed hard label. For a fixed teacher, forward KL subtracts only the teacher entropy, so it has the same student gradients at the same temperature. Distillation →

Changing from natural logs to base-two logs multiplies loss and gradients by 1/ln 2. The optimum stays the same, but the new scale can interact with learning rates and gradient clipping.

The coding interpretation and the exact 2.625 = 1.75 + 0.875 bit example have one owner: Information Theory, cross-entropy and KL →. The implementation section below connects sequence likelihood to masked token loss.

Binary and multilabel targets

A Bernoulli target y ∈ [0,1] has a single positive-class probability q = sigmoid(z). Its binary cross-entropy is the two-outcome log score.

For a hard positive target, −log q diverges logarithmically as q approaches zero. It does not grow exponentially in 1/q. The derivative with respect to q diverges, but the sigmoid-composed logit derivative approaches −1.

For finite logits and hard targets, the loss approaches zero without reaching it. With a soft target, its minimum is the target’s Bernoulli entropy. This is a useful reminder that a correct distribution need not produce zero log loss.

For multilabel tasks, apply one Bernoulli loss per label. This estimates marginal label probabilities. The use of separate losses does not prove that the real labels are independent. Positive-class weights alter the optimal probability under the weighted objective, so calibration needs separate evaluation.

PyTorch: stable BCE with logits ↗

Weighting, calibration and numerical limits

Binary classification uses one Bernoulli output. Multilabel classification uses one per label; single-label multiclass prediction instead uses a normalized categorical distribution. A batch mean averages included elements, not necessarily one loss per example.

Giving positive examples more weight changes the population optimum. Let p be the true positive probability at a fixed input and w > 0 multiply only the positive term. The expected weighted loss is:

For p = 0.20 and w = 4, q* = 0.50. The new output is not calibrated to the original p. At the population optimum, thresholding q* at 0.5 corresponds to thresholding p at $\frac{1}{w+1}$. A larger positive weight lowers that threshold. Actual precision and recall still depend on the fitted model, data and decision rule.

Multiplying every element by one common positive constant only rescales the risk; it does not produce this class-specific shift. PyTorch distinguishes element weights from positive-class weights. Use the intended broadcasting shape and an explicit reduction. BCEWithLogitsLoss contract ↗.

Keep the loss finite without clipping away the error

When q rounds to zero, −log q is infinite. Clipping q to ε caps that penalty. Differentiating through a hard clip also gives zero gradient outside the allowed interval. This changes the training objective; it is not equivalent to stable evaluation from logits.

A wrong extreme logit can still receive a corrective gradient. The logarithm diverges as q approaches zero, while the sigmoid-composed logit gradient stays bounded. Do not describe the probability-space penalty as exponential.

The same score inside an adversarial game

The original GAN discriminator assigns real samples target 1 and generated samples target 0. It maximizes log D(real) + log(1−D(fake)), equivalent to minimizing the corresponding BCE. The minimax generator instead minimizes log(1−D(fake)).

The paper also proposes maximizing log D(fake), or minimizing −log D(fake), to reduce early saturation. These generator objectives are not interchangeable labels for ordinary supervised BCE training. Generative Adversarial Nets, §3 ↗.

Proper scores and Bayes risk

A score S(q,i) evaluates a proposed distribution q when outcome i occurs. Its population risk under p is $L(p,q)=\sum_i p_iS(q,i)$. A proper score makes q = p a minimizer. A strictly proper score makes it the unique minimizer over distributions.

The log score is strictly proper. So is the full multiclass Brier score $S(q,i)=\sum_j(q_j-\mathbf{1}[j=i])^2$. Its Bayes risk is $1-\sum_j p_j^2$.

For alphabets with at least three outcomes, standard regularity conditions severely restrict strictly proper local scores to the log-score form. The binary case has exceptions. Do not turn this qualified result into “cross-entropy is the only proper loss.”

Properness concerns the expected score over probability distributions. It does not guarantee that gradient descent on a deep network finds those probabilities.

The Bayes risk is concave because it is the pointwise minimum of risks that are linear in the target distribution. At a nondifferentiable point, use a supporting supergradient of the concave risk. A subgradient instead belongs to its negative convex regularizer.

Tsallis, spherical and some Rényi-related constructions give other scoring families. Check the exact sign and parameter range. A family name alone does not establish strict propriety, robustness or good finite-data behavior.

Why a local proper score becomes logarithmic

The log score is not the only proper score. A narrower requirement makes it special: the observed loss uses only the probability assigned to the outcome that occurred.

With two outcomes, the other probability is already determined by the observed probability. Full-vector Brier becomes $F(q_i)=2(1-q_i)^2$. It is a strictly proper, non-logarithmic binary counterexample. With three outcomes, the unobserved mass can be split in different ways, so multiclass Brier is not local in this sense.

The first two forecasts give the same log loss, −log 0.2. Their Brier scores differ. The observed-class probability alone is not enough to calculate multiclass Brier.

Same optimum, different gradients

A bounded loss can reduce the influence of extreme errors. It can also weaken a needed corrective signal. For Brier composed with softmax, the gradient contains the softmax Jacobian $J=\operatorname{diag}(q)-qq^\top$.

At q(A) = 0.001 and target A, log loss has derivative −0.999. Full two-class Brier has derivative about −0.003992. Their loss scales differ, so compare more than raw gradient magnitude. The extra saturation factor remains a structural distinction.

This explains a common trap: selecting a score from its probability-space properties while ignoring the output map. The next construction chooses the map and loss together.

Fenchel–Young: pair the map with the loss

Choose a concave entropy-like function H on the simplex. Define a regularized prediction map by balancing a logit score against H. Define Φ as the maximum value of that optimization.

The matched loss has gradient q(z) − p when differentiable. This is not the same as choosing an arbitrary score and composing it with softmax. For a one-hot target and H(eᵧ) = 0, the loss reduces to Φ(z) − zᵧ.

At α = 1, the usual limiting map is softmax. At α = 2, it is sparsemax. α = 1.5 is a useful intermediate case. Constants in the entropy definition affect scale and temperature; do not mix formulas from incompatible conventions.

Sparse predictions can reduce active output support. Exact zero probability creates infinite log loss for a missed target, however. Sampling semantics, rare-token behavior, vocabulary kernels and evaluation all need attention. A sparse-output training loss does not automatically improve throughput or remain comparable to standard perplexity.

Exact zeros make the active support explicit and can reduce the influence of implausible classes. They do not identify noisy labels: rare language tokens can be correct. Forward KL is infinite if the teacher assigns mass where the student assigns zero. Reverse KL has the opposite support requirement.

Sparse outputs also change sampling and vocabulary-wide threshold calculations. A sparse probability vector does not by itself reduce the cost of computing all vocabulary logits. Evaluate fused kernels, memory and distillation under the chosen map rather than assuming softmax implementations transfer unchanged.

Fenchel–Young losses ↗ · Sparsemax ↗ · Sparse sequence-to-sequence models ↗

Margins and support vector machines

Probability estimation is not the only goal. A linear SVM learns a separating function f(x) = w·x + b for labels y ∈ {−1,+1}. Scaling w and b leaves the decision boundary unchanged, so canonical constraints fix their scale.

When the data are not separable, a soft-margin objective allows violations. Eliminating its nonnegative slack variables gives hinge loss.

Support vectors are examples with active constraints and nonzero dual weights. In the hard-margin case they lie on the margin. In the soft-margin case they can lie inside it or be misclassified. They are not always exactly on its boundary.

The linear convex formulation differs from a neural model with a hinge objective. Its squared weight penalty can give uniqueness in w, but this does not automatically give uniqueness in every parameter, such as b. A large margin is useful under suitable assumptions; it is not a universal guarantee of generalization.

Prediction uses the sign of f. At f=0, choose an explicit tie policy. For nonzero w, perpendicular distance is $\frac{|f(x)|}{\|w\|_2}$. Both margin planes are parallel, so their separation is the sum of their distances from the central plane.

Structural risk minimization balances fit with model complexity. In this formulation, C controls violations relative to squared weight magnitude. Margin-based robustness depends on feature scale, perturbation norm and data assumptions. Convexity alone does not prove uniqueness in every parameter. SVM formulation and implementation conventions ↗.

Loss, normalization and regularization

An L2 residual loss evaluates prediction error. An L2 parameter penalty evaluates the size of model parameters. L2 normalization replaces a nonzero vector with $\frac{x}{\|x\|_2}$. They share a norm but perform different operations.

Why ridge stabilizes a linear solve

Assume the features and targets are centered, or handle an unpenalized intercept separately. Minimize $\|Xw-y\|_2^2+\lambda\|w\|_2^2$. Setting its gradient to zero gives $(X^\top X+\lambda I)w=X^\top y$. Use a linear solver rather than explicitly forming an inverse.

$X^\top X$ is the feature Gram matrix. It is proportional to an empirical covariance only after the appropriate centering and scaling. Ridge does not inherently require an inverse covariance matrix. A Gaussian residual model with covariance Σ instead uses $r^\top\Sigma^{-1}r$; $\Sigma^{-1}$ is the precision matrix. These are distinct uses of a quadratic form.

L1 and L2 are not inverse operations. A smooth squared residual is also not universally more stable to optimize than an absolute residual. Compare the model, solver and data, not just the derivative at zero. Ridge objective and solver conventions ↗.

Negative log-likelihood is not the reciprocal of likelihood. Self-supervised learning needs additional mechanisms to avoid representational collapse. Normalization alone is insufficient. For MoE, an auxiliary balance loss is a separate routing objective; changing it changes how capacity is used. Mixture of Experts →

Language-model implementation

A causal language model predicts the next token, not the current token. Shift logits and labels consistently. Padding, prompt-only regions and packed-sequence boundaries need an explicit supervision mask.

# logits: [batch, time, vocabulary]; tokens: [batch, time]
pred = logits[:, :-1].float()
target = tokens[:, 1:]
valid = supervision[:, 1:].bool()
per_token = F.cross_entropy(
    pred.transpose(1, 2), target, reduction="none")
loss = per_token[valid].mean()  # reject/skip an empty valid set

If ignored labels contain −100, use the loss’s ignore-index handling and build a matching valid mask. For operations that gather target entries manually, replace ignored indices before gathering. Do not let −100 select an unintended vocabulary entry.

# Full-vector Brier without allocating a one-hot target array.
q = pred.softmax(-1)
safe_target = target.masked_fill(~valid, 0)
q_true = q.gather(-1, safe_target[..., None]).squeeze(-1)
per_token = q.square().sum(-1) - 2 * q_true + 1
loss = per_token[valid].mean()  # same empty-set policy

import torch

def brier_from_logits(logits, targets, reduction="mean"):
    q = logits.softmax(dim=-1)
    q_true = q.gather(-1, targets.unsqueeze(-1)).squeeze(-1)
    per_token = q.square().sum(-1) - 2 * q_true + 1
    if reduction == "none":
        return per_token
    if reduction == "sum":
        return per_token.sum()
    if reduction == "mean":
        return per_token.mean()
    raise ValueError(f"Unknown reduction: {reduction}")

A token mean gives longer examples more weight. A mean of sequence means weights each sequence equally. In distributed training, averaging rank-local means is incorrect for a global token mean when valid counts differ. Reduce sums and counts consistently.

For a mean natural-log loss L, perplexity is exp(L). Bits per token are L/ln 2. Comparing perplexity across different tokenizers or evaluation distributions is not a controlled comparison. Track the precise mask, denominator and corpus.

Changing the loss is an experiment

A positive mixture of proper scores remains proper over distributions. This does not make a CE–Brier mixture or a late-training switch automatically beneficial. The gradient scale, saturation behavior and optimizer state still change.

Treat such changes as hypotheses. Compare held-out log loss, calibration, rare-token quality, corruption sensitivity and throughput. Separate clean-data and noisy-data effects. Keep reductions, batch composition and compute budgets controlled.

Measure every term separately when adding regularization. A convenient total loss can conceal a routing penalty dominating language modeling, or a changing token denominator. Inspect gradients as well as scalar values.

The motivating blog proposes strong CE correction early and Brier later to reduce suspected-noise influence. That is an unverified language-model research direction. Compare equal-token/equal-compute CE, Brier-plus-softmax, a matched Fenchel–Young map, fixed mixtures and late switches. Separate deliberate corruption from genuinely ambiguous clean language. Measure calibration, rare-token accuracy, downstream quality, memory, throughput and numerical stability at the real vocabulary size.

Sources and connected notes

Gneiting and Raftery: proper scoring rules ↗ · Source discussion of scoring rules and output maps ↗