math

Vectors and Geometry

Use length, angle and orientation for different questions.

Published 2026-04-10 · Updated 2026-10-06

On this page

Find a section

A vector has both magnitude and direction. A similarity score asks whether two directions align. A distance asks how far their coordinates differ. An oriented area asks which way one vector turns toward another.

These quantities are related, but not interchangeable. The distinction matters for embeddings, geometric algorithms and losses. It also prevents treating unit normalization as a complete training objective.

Magnitude, direction and inner product

\begin{aligned}\lVert x\rVert_2&=\sqrt{\sum_i x_i^2}\\a\cdot b&=\sum_i a_ib_i=\lVert a\rVert_2\lVert b\rVert_2\cos\theta\end{aligned}

The angle needs two nonzero vectors. The Euclidean norm is their distance from the origin.

The vector [3,4] has norm 5. Its direction is [3/5,4/5]. An orthogonal rotation changes coordinates but preserves the norm and dot product. This symmetry suits Euclidean geometry; it does not imply that a learned embedding distribution is isotropic.

The L1 norm sums absolute coordinates. In two dimensions, its unit ball is a diamond; the L2 unit ball is a circle. Those constraint shapes affect optimization, but the shape alone is not a guarantee of robustness or sparsity for every objective.

Normalization keeps direction, not diversity

u=\frac{x}{\lVert x\rVert_2},\quad x\ne0\ \Longrightarrow\ \lVert u\rVert_2=1

The zero vector has no defined direction. State the zero/near-zero policy in an implementation.

Normalization removes radial scale. All positive multiples of x become the same u. It is not centering or whitening, and it does not remove statistical dependencies between embedding dimensions.

If every sample maps to [1,0], every embedding has unit norm but the representation has collapsed. Normalization constrains radius, not the diversity of directions. A useful self-supervised objective needs another mechanism to preserve informative variation.

The original LeJEPA/FaceNet examples motivate bounded embedding geometry, not a proof that normalization alone prevents collapse. Inspect the actual objective, architecture and regularizer. SIGReg reference → addresses distributional constraints rather than only radius.

Dividing by max(‖x‖,ε) bounds amplification near zero, but outputs with ‖x‖<ε are not unit vectors. Adding ε inside a norm also changes the map. Do not silently claim exact normalization for those policies.

Distance between unit embeddings

\begin{aligned}u\cdot v&=\cos\theta\\\lVert u-v\rVert_2^2&=\lVert u\rVert_2^2+\lVert v\rVert_2^2-2u\cdot v\\&=2-2\cos\theta\end{aligned}

Both u and v must have unit norm. Their squared distance lies in [0,4].

Three reference directions
Relation	Cosine	Squared distance	Distance
Same direction	1	0	0
Orthogonal	0	2	$\sqrt2$
Opposite direction	−1	4	2

Cosine ranking and Euclidean-distance ranking agree for exactly normalized embeddings. They need not agree when vector magnitudes vary. A high dot product can then reflect large norms rather than closer directions.

Even on a sphere, geometric closeness is only as semantically useful as the learned representation. Unit norm does not establish meaningful neighbors.

Norm, squared loss and penalty

Specify the formula instead of only saying L2
Quantity	Formula	Question
Magnitude	$\lVert x\rVert_2$	How long is one vector?
Distance	$\lVert\hat y-y\rVert_2$	How far is a prediction from its target?
Squared error sum	$\sum_i(\hat y_i-y_i)^2$	How much squared prediction error?
Mean squared error	$\frac1n\sum_i(\hat y_i-y_i)^2$	Average error under a specified reduction
Squared parameter penalty	$\lambda\lVert w\rVert_2^2$	How large are model parameters?

Norm and squared norm have different derivatives. For a nonzero vector, $\nabla\lVert x\rVert_2=\frac{x}{\lVert x\rVert_2}$ . The norm is not differentiable at zero. By contrast, $\nabla\lVert x\rVert_2^2=2x$ everywhere. The common smooth L2 penalty uses the squared norm.

Squared residuals weight large errors more than absolute residuals. MSE therefore has different outlier sensitivity from MAE. Nonsmooth L1 objectives are not automatically unstable; subgradient and proximal methods can optimize them well.

L1 penalties can produce exact zero coefficients in appropriate regularized models. Squared L2 penalties usually shrink continuously rather than select exact zeros. This is a penalty effect, not a universal statement about L1 distances between data points.

Losses and Objectives → owns estimator choice, probability scoring, margins and reduction. Geometry supplies the quantities those objectives use.

Which uniqueness claim is valid?

The Euclidean unit ball is strictly convex, but the norm function is not strictly convex along positive rays. For a nonzero x, the midpoint norm $\left\lVert\frac{x+2x}{2}\right\rVert_2$ equals the average norm $\frac{\lVert x\rVert_2+\lVert2x\rVert_2}{2}$ . Therefore “L2 always gives one optimum” is not a valid claim.

The squared Euclidean norm is strictly convex in its own vector argument. The objective $\lVert Aw-y\rVert_2^2$ is strictly convex in w only when A has full column rank. If A has a nullspace, multiple weights can give the same prediction.

Adding $\lambda\lVert w\rVert_2^2$ with λ>0 makes this fully penalized linear least-squares problem strictly convex. A neural network's nonlinear parameterization remains a different optimization problem. A positive penalty does not make arbitrary neural training globally convex.

Gaussian error and ridge precision

\begin{aligned}-\log p(y\mid\mu,\Sigma)&=\tfrac12(y-\mu)^\top\Sigma^{-1}(y-\mu)\\&\quad+\tfrac12\log\det\Sigma+\tfrac n2\log(2\pi)\end{aligned}

For y, μ in ℝⁿ and positive-definite Σ. With fixed

\Sigma=\sigma^2 I

, optimizing μ is equivalent to minimizing squared Euclidean residuals. A learned or anisotropic covariance changes the objective.

Squared error's Gaussian connection is conditional, not an identity with all likelihoods. If σ also changes, its scale and log-normalization term matter. Negative log density is not the numerical reciprocal of a probability.

\begin{aligned}\min_w\;&\lVert Xw-y\rVert_2^2+\lambda\lVert w\rVert_2^2\\(X^\top X+\lambda I)w&=X^\top y\end{aligned}

Solve the linear system. The displayed λ assumes a sum-of-squares objective; a mean reduction changes its scaling.

Ridge adds λI to the feature Gram matrix, not indiscriminately to any covariance before inversion. A Gaussian parameter prior gives a quadratic penalty with its precision matrix. That precision is inverse prior covariance.

“Inverse loss” is not a standard replacement name for L2. Nor is L2 the dual of L1: the dual norm of L1 is L∞, while L2 is self-dual. Use the exact likelihood, prior and penalty definitions rather than that ambiguous label.

Cross product: oriented area and normal

\begin{aligned}a\times b&=\begin{bmatrix}a_2b_3-a_3b_2\\a_3b_1-a_1b_3\\a_1b_2-a_2b_1\end{bmatrix}\\\lVert a\times b\rVert_2&=\lVert a\rVert_2\lVert b\rVert_2\sin\theta,\quad0\le\theta\le\pi\end{aligned}

The usual cross product here acts on 3D vectors. It is perpendicular to both inputs; its magnitude is parallelogram area.

The right-hand rule fixes direction: curl from a toward b and the thumb gives a×b. Swapping inputs reverses it. For unit x and y axes, [1,0,0]×[0,1,0]=[0,0,1].

The determinant mnemonic has unit vectors i,j,k in its first row and components of a,b below. Expand it to obtain the component formula. The middle component includes the determinant's alternating sign.

Algebraic checks
Property	Identity
Anticommutative	$a\times b=-(b\times a)$
Distributive	$a\times(b+c)=a\times b+a\times c$
Scalar compatibility	$(ka)\times b=k(a\times b)=a\times(kb)$
Self product	$a\times a=0$

Parallel or antiparallel nonzero vectors give zero. A zero input also gives zero. In these cases no unique unit normal exists. Dot product zero tests orthogonality; cross product zero tests collinearity or a zero input.

Alignment is not area

Fixed lengths: |a|=2, |b|=3. Solid blue a points along +x. Dashed burgundy b changes its angle. Shading is the parallelogram they span.

Signed angle 45°

Dot product 4.243Signed z component 4.243Area 4.243Cosine 0.707

Positive orientation: the cross product points along +z.

Exact planar toy with 3D cross-product interpretation. World scale stays fixed. Motion is an angle sweep, not training or a measured representation trajectory.

2D turns and 3D triangle normals

\begin{aligned}\operatorname{orient}(A,B,C)&=(B_x-A_x)(C_y-A_y)\\&\quad-(B_y-A_y)(C_x-A_x)\end{aligned}

With x right and y up: positive means counterclockwise, negative clockwise, zero collinear. Screen coordinates often invert y.

This scalar is the z component of the cross product after embedding the two edge vectors in the xy plane. Its absolute value is twice the triangle area. The sign is useful for turn tests; the magnitude is not an angle.

u, v = B - A, C - A
normal = cross(u, v)
triangle_area = norm(normal) / 2
if norm(normal) > tolerance:
    unit_normal = normal / norm(normal)
# Otherwise the triangle is degenerate at the chosen precision.

Floating-point cancellation can make nearly collinear predicates unreliable. A tolerance depends on coordinate scale and application. Robust computational geometry may need adaptive or exact predicates instead of one arbitrary epsilon.

Use the geometry that matches the task

Graphics uses triangle normals for shading; reversing vertex order reverses the normal. Physics uses r×F for torque and r×p for angular momentum. Geometry algorithms use signed turns to track orientation.

High-dimensional embeddings usually need dot products, norms or distances, not a direct 3D cross product. Projecting them to three coordinates can make a diagram readable, but orthogonality in that projection does not prove orthogonality in the original space.

For embeddings, state whether magnitude carries information before normalizing. For losses, state sum versus mean and squared versus unsquared. For geometry, state axis orientation, coordinate scale and degeneracy policy. These small contracts determine whether the same formula answers the intended question.

Sources and related owners

Cross-product geometry ↗ · Ridge and Gaussian-prior precision ↗