Differentiation and Optimization Geometry

Local change before optimizer choice

f(x+h)=f(x)+\nabla f(x)^\top h+o(\lVert h\rVert)

For a differentiable scalar function, the gradient supplies the first-order linear change.

A derivative is local. A negative first-order prediction does not prove that a large update lowers the loss. Curvature, nonsmooth boundaries and changes in the data can invalidate a distant extrapolation.

Differentiability is stronger than having some coordinate derivatives. The gradient-based linear approximation needs the appropriate regularity. ReLU boundaries and other nonsmooth points require an explicit derivative/subgradient convention.

Partial derivatives and coordinate slices

\begin{aligned}\partial_x f(x,y)&=\lim_{h\to0}\frac{f(x+h,y)-f(x,y)}h\\\partial_y f(x,y)&=\lim_{h\to0}\frac{f(x,y+h)-f(x,y)}h\end{aligned}

Hold the other coordinate fixed. Equivalent notations include ∂ₓf, fₓ and Dₓf.

For $f(x,y)=x^2y+xy^2$ , $\partial_xf=2xy+y^2$ and $\partial_yf=x^2+2xy$ . At (2,3), f=30 and $\nabla f=[21,16]$ . The two partials are the slopes of the surface's coordinate slices.

If y itself depends on x, the total derivative also includes the path through y: $\frac{df}{dx}=\partial_xf+\partial_yf\,\frac{dy}{dx}$ . Holding y fixed and moving along a curve are different experiments.

For $L(w,b)=\frac1n\sum_i(wx_i+b-y_i)^2$ , the two parameter sensitivities are $\partial_wL=\frac2n\sum_i(wx_i+b-y_i)x_i$ and $\partial_bL=\frac2n\sum_i(wx_i+b-y_i)$ . Plain gradient descent computes both at the same old (w,b), then updates them together.

Products add two paths of change

\frac{d}{dx}[f(x)g(x)]=\frac{df}{dx}g(x)+f(x)\frac{dg}{dx}

Leibniz notation names the input variable. The primed form below expresses the same identity.

\begin{aligned}(fg)'&=f'g+fg'\\(uvw)'&=u'vw+uv'w+uvw'\end{aligned}

Differentiate one factor at a time, keeping the remaining factors unchanged in that term.

Think of a rectangle with sides f and g. Its area change is gΔf+fΔg+ΔfΔg. The small corner is second order in the input change, so it disappears after division by the step and a limit.

For x²sin(x), the derivative is $2x\sin(x)+x^2\cos(x)$ . The two terms arise because x reaches the product through two factors. A graph's repeated input therefore needs the sum of both backward contributions.

Compositions multiply local rates

\begin{aligned}(f\circ g)'(x)&=f'(g(x))g'(x)\\\frac{dy}{dx}&=\frac{dy}{dv}\frac{dv}{du}\frac{du}{dx}\end{aligned}

Evaluate each local derivative at its own forward value. Leibniz notation suggests cancellation, but the chain rule is not division of arbitrary symbols.

If temperature changes by 2°C/hour and a sensor changes by .5 units/°C, its reading changes by 1 unit/hour. Multiplying the rates both propagates sensitivity and checks units.

For $h(x)=\exp(\sin(x^2))$ at x=.5: a=.25, b=sin(a)=.247404, h=exp(b)=1.280696. The local rates are 2x=1, cos(a)=.968912 and exp(b)=1.280696. Their product is 1.240883.

A deep model composes vector operations, not only scalar functions. The same rule applies, with Jacobians and sums over intermediate coordinates. Associativity permits efficient grouping; matrix multiplication generally does not commute.

Values go forward; sensitivities come back

The same scalar graph can have one chain or an added edge a→L. An adjoint is ∂L/∂node. All incoming backward contributions must be added.

Output expressionInput x 0.50Execution step 8 / 8

xvalue 0.5000adjoint 1.2409

a=x²value 0.2500adjoint 1.2409

b=sin(a)value 0.2474adjoint 1.2807

c=exp(b)value 1.2807adjoint 1.0000

L=cvalue 1.2807adjoint 1.0000

No extra edge: only c contributes to L.

Backward: a=x² sends 2x times a's total adjoint to x. $\frac{\partial L}{\partial x}=1.2409$ .

Exact scalar illustration, not a trained network. Playback follows operation order; x and output expression remain fixed. Zero adjoints before their propagation mean not yet accumulated, not necessarily a final zero derivative.

Jacobians and vector–Jacobian products

\begin{aligned}J_f&\in\mathbb R^{m\times n},\quad(J_f)_{ij}=\frac{\partial f_i}{\partial x_j}\\\nabla_x(L\circ f)&=J_f^\top\nabla_f L\end{aligned}

Column-gradient convention: n inputs, m outputs. Reverse mode applies a vector–Jacobian product without materializing the full matrix.

At a scalar node v, its adjoint is $\bar v=\frac{\partial L}{\partial v}$ . Reverse traversal applies $\bar u\mathrel{+}=\bar v\,\frac{\partial v}{\partial u}$ for each parent u. The += matters when several children depend on the same u.

\begin{aligned}L&=\exp(\sin a)+a,\quad a=x^2\\\bar a&=\exp(\sin a)\cos a+1\\\bar x&=2x\bar a\end{aligned}

At x=.5, the chain contributes 1.240883 and the direct a→L edge contributes 1. The total is 2.240883.

For a scalar loss with many parameters, reverse mode computes all parameter gradients in one backward traversal. Forward mode instead pushes an input-direction tangent through the graph, producing Jv. Choose the mode by input/output dimension and the requested derivative product.

Reverse mode needs forward values for local derivatives. Saving them costs memory; recomputation trades compute for storage. These choices affect execution, not the derivative identity.

Automatic differentiation and checks

Automatic differentiation evaluates local derivative rules on the executed program. It is neither symbolic simplification of the whole function nor a finite-difference approximation, although floating-point arithmetic still introduces rounding.

A small PyTorch reference computes the chain derivative. This snippet is illustrative; the pure numerical model is tested separately.

import torch
x = torch.tensor(0.5, dtype=torch.float64, requires_grad=True)
a = x.square()
h = a.sin().exp()
h.backward()
# x.grad is approximately 1.2408826092

PyTorch backward ↗ accumulates leaf gradients. Reset or intentionally accumulate them between passes. A nonscalar output needs an output seed of matching shape; summing outputs is one particular scalar objective, not a neutral substitute.

A numerical check uses $\frac{f(x+\varepsilon)-f(x-\varepsilon)}{2\varepsilon}$ . Compare several step sizes in high precision away from nonsmooth points. Large ε has truncation error; very small ε has cancellation. For vectors, a directional check compares that quotient along d with $\nabla f\cdot d$ .

Detached tensors, in-place modification of saved values, ignored branches or custom backward rules can change what the program differentiates. A plausible loss curve is not a derivative correctness test.

Steepest direction depends on the norm

The gradient collects the coordinate sensitivities into one column vector. Its Euclidean norm combines those components into a length.

\begin{aligned}\nabla f(x)&=\left[\frac{\partial f}{\partial x_1},\ldots,\frac{\partial f}{\partial x_n}\right]^\top\\\lVert\nabla f(x)\rVert_2&=\sqrt{\sum_{i=1}^n\left(\frac{\partial f}{\partial x_i}\right)^2}\end{aligned}

These definitions need differentiability for the first-order directional interpretation. A coordinate gradient is not a coordinate-invariant measure of sensitivity.

\begin{aligned}D_df&=\nabla f\cdot d\le\lVert\nabla f\rVert_2\lVert d\rVert_2\\d_{\mathrm{ascent}}&=\frac{\nabla f}{\lVert\nabla f\rVert_2},\quad\lVert d\rVert_2=1,\ \nabla f\ne0\end{aligned}

Cauchy–Schwarz identifies the steepest first-order ascent under a Euclidean unit-step constraint.

Negative normalized gradient gives Euclidean steepest descent. A different norm, metric or parameter scaling can give a different steepest direction. This distinction matters when comparing preconditioned updates.

For $\nabla f=[3,4]$ , the norm is 5, unit ascent is [.6,.8] and unit descent is [−.6,−.8]. The norm is the largest directional derivative among Euclidean unit directions, not a promise that every small displacement changes f by that amount.

At zero gradient there is no unique steepest first-order direction. The zero vector is the gradient, not a defined unit direction. Momentum, stochasticity or higher-order structure can still produce movement; not every optimizer automatically stops.

A descent direction is not a safe finite step

\begin{aligned}\theta_{t+1}&=\theta_t-\eta g_t\\\lVert\Delta\theta\rVert_2&=\eta\lVert g_t\rVert_2,\quad\eta\ge0\end{aligned}

For plain gradient descent. Adaptive scaling, momentum and clipping modify the actual update.

A large gradient with a large learning rate can overshoot. Small gradient norm can mean a useful stationary region, bad scaling, cancellation or a plateau. It does not distinguish these cases on its own.

For $f(x)=\frac12\lambda x^2$ with λ>0, the update is $x'=(1-\eta\lambda)x$ . This scalar quadratic converges when $0<\eta<\frac2\lambda$ . At the upper boundary it oscillates without reducing magnitude; above it, magnitude grows. Gradient direction is unchanged, but finite-step stability is not.

g_{\mathrm{clip}}=\begin{cases}g,&\lVert g\rVert_2\le\tau\\\dfrac{\tau}{\lVert g\rVert_2}g,&\lVert g\rVert_2>\tau\end{cases},\quad\tau>0

At zero g, retain zero without dividing. This clips gradient norm, not necessarily the final preconditioned update.

Clipping preserves a nonzero gradient's direction while reducing its magnitude. It can prevent a large update, but it does not correct NaNs, a wrong derivative or an unsuitable objective. Gradient Clipping reference → retains the training-specific implementation conditions.

Curvature classifies stationary points

\begin{aligned}H_{ij}&=\frac{\partial^2f}{\partial x_i\partial x_j}\\f(x+h)&=f(x)+\nabla f(x)^\top h\\&\quad+\tfrac12h^\top Hh+o(\lVert h\rVert^2)\end{aligned}

For a C² function, continuous mixed second partials commute, so H is symmetric.

At an interior stationary point, $\nabla f=0$ removes the linear Taylor term. The Hessian gives the quadratic term, which can vanish in some or all directions. Diagonalize the symmetric Hessian: $h^\top Hh=\sum_i\lambda_i a_i^2$ in an orthonormal eigenbasis.

Second-order classification at a stationary point
Hessian signature	Conclusion
All eigenvalues positive	Strict local minimum
All negative	Strict local maximum
Positive and negative	Saddle
Some zero, no definitive mixed-sign test	Higher-order analysis may be needed

Positive curvature away from a stationary point does not mean the current point is a minimum. Positive semidefinite curvature at a stationary point is necessary for a smooth local minimum, but it is not sufficient. x⁴ has a strict minimum with zero second derivative; −x⁴ has a maximum with the same zero second derivative.

One negative eigenvalue rules out a smooth local minimum. To call a nondegenerate critical point a saddle rather than a maximum, it must also have positive curvature. A constrained boundary requires a different stationarity and feasible-direction analysis.

One point, several possible directions

Every example is stationary at the origin. Horizontal axis: displacement r from −1 to 1. Vertical axis: Δf on a fixed scale, with the zero line through the burgundy origin.

LandscapeSlice direction 90°

Hessian eigenvalues 1, −1

d^\top Hd

−1.000Δf at r=1 −0.500

Negative curvature: moving away from zero along this slice lowers f.

Exact toy functions. Angle sweeps are not optimizer trajectories. A flat quadratic slice in the saddle does not make the entire Hessian zero.

Index and coordinate invariance

\operatorname{index}(H)=\#\{i:\lambda_i<0\}

At a nondegenerate critical point: index 0 is a minimum, N a maximum, and intermediate indices are saddles.

For f(x,y)=xy, the Hessian is [[0,1],[1,0]]. Both coordinate-axis curvatures are zero, yet its eigenvalues are ±1. With $u=\frac{x+y}{\sqrt2}$ and $v=\frac{x-y}{\sqrt2}$ , $f=\frac{u^2-v^2}{2}$ . Mixed terms can hide an escape direction.

An invertible linear coordinate change preserves the positive/negative/zero counts of a quadratic form. Eigenvalue magnitudes need not remain the same under arbitrary scaling. At a stationary point, smooth local coordinate changes preserve this signature; away from stationarity, extra Hessian transformation terms matter.

High dimension is not a landscape model

Two different claims must stay separate: critical points occupy little ambient volume, and minima are uncommon among critical points. For a Morse function, nondegenerate critical points are isolated. This volume-zero fact already appears in one dimension; it does not establish saddle dominance.

The second question needs a sampling model over functions or critical points. Merely writing $P(\mathrm{minimum}\mid\nabla f=0)$ does not specify it. Gradient-zero events can have probability zero at a fixed random location, so a conditional density or point-counting construction is needed.

P(\lambda_1>0,\ldots,\lambda_N>0)=2^{-N}

Independent, unbiased eigenvalue-sign toy. This is an intuition, not a random-matrix law. Real symmetric-matrix eigenvalues are correlated.

A minimum requires every curvature direction to be nonnegative, and every eigenvalue positive in the nondegenerate case. Mixed signs supply both up and down directions. But dimension alone determines neither sign distribution nor optimizer behavior.

For example, $f(x)=\|x\|_2^2$ has exactly one critical point, its global minimum, in any dimension. It has no saddles. This strongly convex example rules out a universal dimension-only claim.

What the GOE result actually says

The Gaussian Orthogonal Ensemble (GOE) models a centered real symmetric random matrix with Gaussian entries and an appropriate normalization. Its limiting eigenvalue density is symmetric around zero, so a large centered matrix usually has many eigenvalues of each sign.

P(H_N\succ0)=\exp\{-cN^2+o(N^2)\},\quad c>0

An asymptotic positive-definiteness event for centered GOE matrices, not the probability that a trained network reaches a bad minimum.

Dean and Majumdar ↗ report the Gaussian-ensemble N² exponent. It describes a specified matrix ensemble. A landscape Hessian can correlate with the loss value and gradient; conditioning on a low value can shift its spectrum upward.

Do not turn a vanishing fraction into zero absolute count. A landscape can have increasingly many minima while saddles grow faster. Nor does uniformly counting critical points match how an optimizer visits their attraction regions.

Counting points requires conditional weighting

\begin{aligned}C_k(D)&=\#\{x\in D:\nabla f(x)=0,\ \operatorname{index}H(x)=k\}\\\mathbb E[C_k(D)]&=\int_D\mathbb E\bigl[|\det H(x)|\,\mathbf1\{\operatorname{index}H(x)=k\}\\&\hspace{5em}\mid\nabla f(x)=0\bigr]\,p_{\nabla f(x)}(0)\,dx\end{aligned}

Kac–Rice under suitable random-field regularity/nondegeneracy assumptions.

p_{\nabla f(x)}(0)

is a density, not the mass of an exact zero event.

The gradient density locates candidate zeros; the determinant weights their local Jacobian volume change; the indicator selects an index. Counting minima sets k=0, while nondegenerate saddles sum k=1 through N−1.

This determinant-weighted conditional expectation explains why sampling an unconditional GOE matrix is not the same as sampling a typical landscape critical point. Gaussian field and spin-glass calculations can use random-matrix tools, but their conditioning and energy regimes remain part of the result.

Morse theory organizes the local geometry

A Morse function has nondegenerate critical points. The Hessian's invertibility isolates such a gradient zero locally. This is a structural statement, not a probability estimate.

The Morse lemma gives local coordinates with k negative squares and N−k positive squares at an index-k point. It explains why index is the invariant. It does not by itself prove that saddles outnumber minima.

The linked landscape article also connects index to topology as a level set crosses a critical value. That reference is useful for the topological extension; the optimization argument here remains conditional on the landscape model.

Translate geometry into training diagnostics

Real neural losses have parameter symmetries, degenerate directions, structured data and nonsmooth operations. A random-Hessian model can provide intuition without being an empirical diagnosis of a particular run.

Observed signal is not a diagnosis
Signal	What it can support	What else to inspect
Small gradient norm	Small local first-order change in this metric	Loss scaling, clipping, update norms, curvature
Large norm	Large possible Euclidean directional rate	Batch reduction, outliers, step size, numerical errors
Negative Hessian direction	A local escape direction at stationarity	Magnitude, feasible directions, optimizer noise
Near-zero eigenvalues	Weak quadratic curvature	Higher orders, symmetry and parameterization
Training loss decreases	The chosen objective improved on these batches	Validation, generalization and actual update behavior

Separate full-objective gradients from minibatch estimates. Stochastic gradients need not shrink to zero near a population optimum. Per-parameter rescaling can also change gradient norm without changing the represented function.

Gradient-norm training reference → and batch/LR reference → own run-specific scaling and update choices. Losses and Objectives → defines what those derivatives measure.

Sources and related owners

Hessians, random landscapes and Morse points ↗ · Gaussian-matrix large deviations ↗ · Autograd backward ↗