Self-Supervised Vision Learning

What should a reusable visual backbone preserve?

A reusable backbone must preserve information that several later tasks can use. Recognizing an entire image and locating one part are related, but different, requirements. A global representation can support classification while losing the spatial distinctions needed for segmentation or depth.

Self-supervised learning (SSL) creates training targets from the data rather than requiring a task label for every image. The target can be pixels, prototype probabilities or another view’s features. That choice determines which differences the encoder is asked to retain.

There are two separate questions: what should related views agree on, and what prevents every image from receiving the same representation? Start with the target contract. Then inspect the anti-collapse mechanism. Finally, evaluate the exact feature interface you intend to reuse.

Separate image features, patch features and loss heads

For a batch of B images with P patches, a transformer can expose patch features [B,P,D] and one global class-token feature [B,D]. D is the backbone width. A projection head can map those features to K prototype logits, giving [B,P,K] or [B,K].

Prototype probabilities are training targets, not automatically calibrated semantic labels. A prototype can organize visual similarity without meaning a named class such as “wheel.” The downstream application can use backbone features rather than the prototype head.

Keep spatial correspondence explicit. Same-view teacher and student patch positions can match directly when the student masks some inputs. Crops of different locations require coordinate correspondence before a patchwise comparison. A global cross-view loss does not need that same patch-index identity.

Choose what the model predicts

Do not infer that pixel targets make dense features universally weak, or that global distillation has no target. Compare the complete architecture, training scale and evaluation. MAE ↗ and BEiT ↗ illustrate the two reconstruction contracts.

iBOT: learn an online visual tokenizer

iBOT means Image BERT Pre-Training with Online Tokenizer. A teacher generates patch-level distributions as training proceeds. The student predicts those targets from a masked version of the same view. There is no separate fixed-tokenizer pretraining stage.

A second objective aligns class-token predictions across views. This global objective supports semantic organization of the online targets. The patch objective adds a local prediction requirement. The two losses answer different questions rather than duplicate each other.

“Online” describes how targets evolve. It does not mean that the teacher receives the same backward update as the student. iBOT ↗ uses a moving teacher and stopped target predictions.

Categorical patch targets, not cosine regression

The patch loss keeps same-view spatial correspondence. The class-token loss instead asks different crops to agree globally. Their weights set a tradeoff between local prediction and global agreement; they do not make the two targets identical.

For a selected patch, the logit derivative is (pₛ−qₜ)/(|M|τₛ). A positive component lowers a student logit under gradient descent; a negative component raises it. The target distributes credit across prototypes instead of selecting only one hard code.

Use stable log-softmax, not the logarithm of an already underflowed probability. Normalize teacher rows to sum to one. Specify whether averaging occurs per image, per view or across all selected batch tokens. Those choices change sample weights when mask counts differ.

If M is empty, do not divide by zero. The teaching model explicitly returns zero loss and gradient. A real recipe can instead require at least one masked token. This boundary must be declared.

Input masking and loss selection are different operations

The input mask stays fixed when the loss-position switch changes. In masked-only mode, changing visible patch1’s assigned logit does not change this local loss. In all-token mode, that prediction receives its own direct target.

The table shows derivatives with respect to independent output logits. A real transformer couples outputs through context. Visible patch embeddings can affect masked predictions and receive indirect gradients even without a direct loss at their own positions.

A stopped target and a moving teacher

Compute teacher targets without retaining a loss gradient through them. Compute student predictions and the declared loss. Backpropagate through the student, update student parameters, then apply the recipe’s teacher update. Keep the teacher’s normalization and state updates explicit.

Momentum controls target lag. An EMA teacher is not a universally sufficient anti-collapse proof. The full view, temperature, centering and objective recipe matters. A frozen larger teacher used for model distillation is another contract; it does not follow the student by EMA. Distillation covers that broader transfer setting.

Centering, sharpening and prototype allocation

Centering changes teacher prototype allocation across samples. Sharpening changes each target distribution’s concentration through temperature. These are not the same operation. Nor is centering prototype outputs equivalent to whitening backbone features.

Original iBOT investigates centering and sharpening for patch targets as well as its DINO-style global targets. It is therefore incorrect to describe iBOT as requiring neither. DINOv2 uses three Sinkhorn–Knopp normalization iterations for teacher outputs, while the student uses softmax.

Batch allocation can discourage one prototype from absorbing every sample. It does not prove that each prototype is a named semantic category. Finite batches, distributed aggregation and numerical precision affect the implemented normalization. Preserve the actual target preparation when reproducing a method.

Shared or separate heads depends on the recipe

Original iBOT shares projection-head parameters between class and patch tokens. Its ablation supports sharing in that setup. DINOv2 instead uses separate DINO and iBOT heads after finding untying helpful at scale.

These choices are not contradictory laws. The global and local objectives can need different loss-space mappings. Backbone sharing, head sharing and teacher sharing are independent decisions. Record which parameters each objective updates rather than summarize the whole method as simply “shared.”

Keep pretraining and evaluation conditions attached

The second result updates the backbone and uses a higher evaluation/training resolution. It is not a frozen-feature result. Neither number establishes universal dense-task quality. Detection and segmentation need their own protocols. See iBOT Tables 1–3 ↗ for the specific comparisons.

DINOv2: make global and local training scale together

DINOv2 combines global DINO cross-entropy, masked-patch iBOT cross-entropy and a feature-spreading regularizer. It scales this recipe with curated data, a large teacher and smaller distilled models. Frozen global and patch features are both intended outputs.

Sinkhorn normalization, separate heads, residual stabilization and efficient crop execution support that larger run. A single “DINOv2 loss” label hides these different responsibilities. A reproduction needs the full training contract, not only a sum of three symbols. DINOv2 ↗ describes the combined recipe.

Spread features without calling them Gaussian

Close neighbors receive a large penalty. Minimizing the negative log distance encourages the batch features to spread. The batch, normalization and nearest-neighbor definition are part of the mechanism.

Identical features give zero distance, so the logarithm is not finite. Practical implementations need an explicit epsilon or equivalent numerical policy. B must permit a non-self neighbor. Changes in batch composition change the nearest-neighbor set.

KoLeo is based on differential-entropy estimation. It is not the same as fitting an isotropic Gaussian. Unit-normalized features live on a sphere, whereas a Gaussian has variable radius. The geometry and target constraints differ.

Preserve crop boundaries while packing computation

Multi-crop learning mixes long global-crop sequences and shorter local-crop sequences. Packing concatenates those sequences into one compute stream. A block-diagonal attention mask prevents tokens in one crop from reading another crop. Concatenation without that boundary changes the learning problem.

Efficient stochastic depth skips the dropped residual computation rather than calculating it and multiplying the output by zero. DINOv2 uses a 40% drop rate. The saved fraction concerns those residual computations; it is not a universal 40% reduction in total training FLOPs.

LayerScale initializes residual branches with small learned diagonal multipliers. Its role is residual stabilization, not target normalization. Fused kernels, sharding and attention layouts address execution cost; they do not replace the global/local objectives. Distributed Training and Parallelism owns sharding and synchronization.

Read feature visualizations as qualitative evidence

Patch-feature PCA can reveal foreground structure and part correspondences across images. DINOv2’s examples include related regions despite pose or appearance differences. Successive components can organize parts such as eyes, wings or wheels.

PCA components have arbitrary sign. A threshold chosen for selected images is not a guaranteed object mask on every input. A visually convincing feature map is useful qualitative evidence, not a segmentation score.

To use a feature visualization, state the feature layer, preprocessing, PCA fit set, component selection and threshold. Fit and evaluate learned readouts separately. Otherwise a selected image can hide failures or preprocessing dependence.

iBOT++: supervise visible predictions too

Masked prediction supplies direct targets only at selected masked positions. Visible tokens can support reconstruction without being directly anchored to their corresponding teacher predictions. That distinction motivates iBOT++ within TIPSv2.

All-token supervision changes both the selected positions and, under the mean convention here, the denominator. It does not mean “remove masking from the input.” Removing masking during a separate distillation stage is another choice.

The proposed explanation is stronger anchoring of local semantics. The tested improvement belongs to the full training and evaluation setup; all-token supervision is not guaranteed to help every target, augmentation or domain.

Patch-text alignment needs a text learning path

A visual teacher target is not itself a text embedding. iBOT++ matches visual prototype distributions. TIPSv2 also has an image-text learning path. That shared training context is what makes patch-text alignment a meaningful capability to evaluate.

For zero-shot segmentation, compare patch features with embeddings of class names and assign spatial labels under a declared upsampling/inference protocol. This is different from fitting a supervised segmentation head.

A distilled student can outperform its teacher on a particular feature use. Changed initialization, loss positions and optimization can reorganize representations. That observation is not evidence that all students universally surpass teachers.

Head-only EMA and captions at different granularities

TIPSv2 combines all-token self-distillation, head-only EMA and captions at multiple granularities. The teacher and student share the vision encoder in the head-only design, but the target head remains an EMA copy. This avoids storing a second EMA encoder.

The combined image-text objective supplies another constraint on the encoder. Do not transplant head-only EMA into an SSL-only recipe and assume the same stability. Removing head EMA too is yet another configuration.

Synthetic captions from PaliGemma and Gemini provide different levels of detail alongside web text. More detailed captions are not automatically better alone. Sampling across granularities changes which visual distinctions the model must use. This is a data/objective design choice, not simply a longer-caption rule.

One ablation is not every downstream task

The iBOT++ change adds 14.1 mIoU points in this experiment. The cumulative ablation uses 100k steps at 224 resolution; it is not the final high-resolution model comparison. Image-only segmentation does not improve in every ablation row.

The paper evaluates 9 tasks across 20 datasets with frozen encoders. Those tasks include global and dense image-text uses as well as image-only uses. A supervised readout and a zero-shot text protocol do not measure the same contract. TIPSv2 Tables 3–4 and evaluation protocol ↗ give the conditions.

LeJEPA: agreement plus distribution regularization

Agreement objectives can be simple, but their collapse safeguards can become complicated. LeJEPA combines a centered multi-view squared-distance objective with Sketched Isotropic Gaussian Regularization, or SIGReg.

The global-view center remains differentiable. Changing one global embedding changes both its own prediction and the center used by other views. A stop-gradient center implements a different objective.

The core loss can be concise—approximately 50 lines in the paper’s presentation. Data loading, augmentation, model definition, distributed correctness and evaluation still require a full training pipeline. Simpler loss code does not remove those contracts. LeJEPA ↗ develops the approach.

View agreement alone admits collapse

A constant encoder can make every distance zero. A low agreement loss therefore does not by itself show useful representations. A second term must constrain representation spread or distribution. The example changes within-image agreement and between-image separation independently.

Centered agreement is not identical to pairwise agreement

Be precise when replacing pairwise global-to-all distances with distance to the mean. For any set of global vectors g and comparison vectors a, the exact identity contains an additional variance term.

For global and all-view scalars −1,+1, the centered loss is 1 while the pairwise loss is 2. The missing term is 1. The LeJEPA v3 paper’s Eq6→7 and Appendix B.6 omit this term. The centered loss shown in its implementation is a valid declared objective, but not an exact equality to that pairwise loss.

This algebraic distinction does not invalidate reported experiments using the implemented centered objective. It matters when deriving gradients or writing a reproduction from the pairwise formula. LeJEPA v3, Eq6–7 and Appendix B.6 ↗ provides the version-specific derivation.

Why isotropy needs a scale constraint

An isotropic covariance treats every direction equally. But its scale must be fixed. A distribution with covariance sI is isotropic for any positive s; only s=1 gives identity covariance.

Gaussian matching goes beyond a covariance penalty. Means, variances and pairwise covariances can all match the target while higher-order structure differs. The next example isolates that failure rather than relying on a high-dimensional picture.

What the Gaussian optimality argument does—and does not—prove

The Fisher information functional measures the expected squared magnitude of the density’s score gradient, $\nabla\log p$. Here “score” refers to a density derivative, not a prediction confidence.

With a fixed covariance trace T, minimizing $\sum_{j=1}^D\frac1{\lambda_j}$ favors equal positive eigenvalues. By Cauchy–Schwarz, that sum is at least $\frac{D^2}{T}$. This gives the isotropic Gaussian as the minimizer of this functional under the constraint.

The paper connects this geometry to downstream risk through specified task and estimator models, including linear-probe sensitivity and asymptotic nonlinear-estimator bias. Those assumptions matter. Distribution shape alone cannot prove that an encoder retains the labels needed for every possible task.

For example, an encoder could preserve the desired marginal shape while discarding one task-relevant distinction. View construction and feature information remain essential. The argument motivates a regularizer; it is not an unconditional all-task optimality certificate.

A joint distribution is more than its coordinate marginals

Cramér–Wold relates a joint distribution to all one-dimensional linear projections. Checking only the coordinate axes is weaker. Sampling a finite set of random directions is also weaker than checking every direction.

Let $U\sim\mathcal N(0,1)$ be a standard Gaussian and $R\in\{-1,+1\}$ an independent fair sign. Define $Z=(U,RU)$. Both coordinates are standard Gaussian, and covariance is identity. Yet Z lies on two diagonals, not throughout the plane.

The 45° projection is $\frac{U(1+R)}{\sqrt2}$: half its mass is exactly zero and half is a Gaussian with variance 2. Its total variance remains 1. This singular toy law illustrates the insufficiency of marginal/covariance checks; it is not a counterexample to a theorem requiring a smooth density.

Rotate a direction that reveals the hidden difference

First compare the diagonal law at 0° and 45°. Then compare it with the isotropic Gaussian. Rotation changes the projected distribution while leaving the joint law fixed. No encoder update is performed.

Characteristic functions expose more than covariance

Near zero, the characteristic function reflects low-order moments when those moments exist. Other frequencies expose further distributional structure. A symmetric zero-mean target has no imaginary component, but an empirical sample can still have one.

Condition on the fair sign in the diagonal example. With sign −1, the 45° projection is zero and its characteristic function is one. With sign +1, the projection is $\sqrt2\,U$ and has characteristic function $e^{-t^2}$. Average the two equally likely branches:

Both branches together give zero mean and variance one. Those first two moments match the standard Gaussian, but the full signature differs. A covariance-only check cannot express that difference.

Estimate the signature from a batch

Unit normalization is necessary: a projection of $\mathcal N(0,I)$ onto a vector a has variance $\|a\|_2^2$. Without normalization, the standard-normal target is wrong.

Random directions form a stochastic approximation. Resampling increases directional coverage across updates, but a finite run is not a proof of full Gaussianity. Use reproducible synchronized direction sampling when combining evidence across workers.

Make the frequency integral and its scale explicit

The scene uses 17 knots on [0,3], doubles the positive half and gives trapezoid endpoints half weight. The paper also presents 17 knots on [−5,5]. Integration limits, weighting and normalization are configuration choices; do not mix the formulas and keep the same λ as if the scale were unchanged.

For real samples, the negative-frequency characteristic value is the conjugate of the positive one. With a symmetric real target and window, the squared discrepancy is even. That justifies positive-half doubling, not omission of endpoint weights.

Across workers, combine cosine and sine sample sums and counts before squaring their difference from the target. Averaging already squared worker losses gives a different result. Unequal worker counts require a count-weighted mean.

Finite tests have sampling noise and blind spots

Even when the population matches the target, the finite-batch discrepancy generally remains positive at nonzero frequencies. Increasing B reduces that sampling contribution. Multiplying by B changes its scale rather than making it disappear.

A zero discrepancy on selected frequencies and directions does not prove that all untested projections match. Conversely, a positive empirical loss does not alone prove a population mismatch. A calibrated hypothesis test needs its null reference and significance rule; a training loss is not automatically a p-value.

Finite sketches trade coverage for cost. A 256-direction setting is an example, not a Cramér–Wold guarantee. Analyze more directions, different seeds and held-out projections before drawing distributional conclusions.

Linear complexity means fixed slices and frequencies

SIGReg is linear in D when M and T are held fixed. It is not independent of slice count or batch size. Chunking frequency or sample dimensions can reduce temporary memory, with corresponding reduction and autograd contracts.

The covariance comparison describes a direct implementation, not a claim that every other anti-collapse algorithm necessarily has the same cost. Measure the actual kernel and communication path. A compact equation is not a latency measurement.

Two small loss contracts with explicit reductions

This patch-loss sketch assumes student logits [B,P,K], normalized stopped teacher targets of the same shape, and a Boolean selected mask [B,P]. Empty selection needs the explicit policy shown. It does not replace target preparation or teacher updates.

q = teacher_targets.detach()
log_p = log_softmax(student_logits / tau_s, dim=-1)
ce = -(q * log_p).sum(dim=-1)  # [B, P]
count = selected.sum()
if count > 0:
    patch_loss = ce[selected].sum() / count
else:
    patch_loss = student_logits.sum() * 0

For centered agreement, keep both global embeddings and their center in the gradient graph. This sketch uses [V,B,D], the first G global views and a mean over coordinates as well as views/images.

z = encode_views(views)  # [V, B, D]
center = z[:G].mean(dim=0)  # [B, D], no detach
agreement = (z - center[None]).square().mean()
regularizer = mean(sigreg(z[v]) for v in range(V))
loss = (1 - lambd) * agreement + lambd * regularizer

These are teaching pseudocode, not benchmarked training implementations. Test masks, target-row sums, same-view correspondence, precision, gradients and reductions. Use the actual distributed autograd implementation when feature statistics communicate between workers.

Pretrained, joint and encoder-free visual paths

A useful visual backbone can enter a language system through several paths. The paths differ in training order and gradient ownership, not simply in model size.

DeepSeek initially freezes the encoder except final normalization, keeps its projector trainable, then unfreezes at the language-learning-rate decay with a smaller rate. Its high-resolution bootstrap trials added little final quality in the reported later-stage pipeline. That is stage-specific evidence, not a ban on high-resolution pretraining.

K3 reports that joint from-scratch training matches its SigLIP-initialized baseline with fewer gradient spikes. Its mixture includes captions, interleaved documents, OCR, video, perception and rendered code artifacts. Gemma’s encoder-free experiment is not an equal-compute proof that raw patches always beat frozen encoders.

Keep these source-specific recipes connected to Frontier Model Training Recipes and Small and On-Device Model Recipes. They are applications of visual representations, not additional SSL objectives. Compare the source-specific DeepSeek report §4.2 ↗, Kimi K3 report ↗ and Gemma 4 report ↗.

Resolution and merging set the language-side token cost

Before merging, an image with height H, width W and patch size p has approximately (H/p)(W/p) spatial tokens, subject to processor rounding. Grouping s×s tokens divides that count by s² when the grid is divisible. Channel width or a learned merger must absorb the grouped information.

GLM-5.3-Flash’s reported vision encoder has 24 blocks, width 1024, 16 heads of 64 dimensions and FFN width 4096. It uses 2×14×14 temporal/spatial tubelets and a 2×2 spatial merge. A 448×448 image gives $32\times32=1024$ patch positions, then $16\times16=256$ language-side positions per temporal tubelet, before processor-specific handling.

The vision blocks use non-causal attention, QK normalization, spatial-temporal positions/RoPE, pre-RMSNorm and clamped SwiGLU. After normalization and merge, a projection reaches language width 4096 with a gated intermediate width 10240. These are released-architecture details, not evidence for a particular SSL pretraining loss.

K3 uses 2×2 token grouping and supports images up to 3584×3584. DeepSeek uses 3×3 pixel unshuffle for a 9× token reduction. Gemma’s frozen encoder path chooses among 70,140,280,560 and 1120 output tokens while preserving aspect ratio. Those token limits are not interchangeable evaluation settings.

Compare the vision-encoder token count with the language-side count separately. Compressing after the encoder does not erase all encoder-side attention cost. The text decoder’s KV cache and prefill cost see the final injected sequence. The GLM-5.3-Flash configuration ↗ records the released architecture; it does not disclose the full training recipe.

A visual input interface is not an SSL recipe

An image-capable service does not disclose a backbone pretraining recipe. Apple’s AFM3 announcements establish multimodal/image-input interfaces and a distinct image-generation service, but do not provide a reconstructible vision SSL objective.

Shieldstral inherits a Pixtral vision encoder from Ministral. Its approximately 4.5M multimodal constructed examples, 14-category taxonomy, query variations and asymmetric reranker filtering concern policy classification. Whether and how the vision components are updated remains undisclosed. Structured Decision and Safety Models retains the full safety/data recipe.

Likewise, GLM’s reported 30T multimodal exposure does not reveal a complete modality ratio, deduplication or optimizer schedule. Render–inspect–revise agent trajectories train a different behavior from image-only SSL. An architecture configuration and a marketing evaluation cannot fill in an unknown objective.

Evaluate the feature contract you intend to use

Select the intended interface before selecting a headline number. A frozen global vector, frozen spatial tokens, a fitted readout, a fine-tuned encoder and an end-to-end language system can give very different answers.

Hold augmentation, model size, exposure, optimizer budget and evaluation constant when testing a proposed loss change. Check feature variance, nearest neighbors and held-out direction discrepancies as diagnostics, not replacements for downstream evaluation.

PCA is qualitative. Training-loss correlation can support checkpoint selection within its tested setting; it is not a universal ranking across domains and objectives. Compare the actual readout contract and its uncertainty. Pretraining: Experiments and Run Design covers experiment design; Detection and Spatial Predictions covers a downstream spatial-selection boundary.

Related branches, not one mandatory sequence

DINO (2021) develops global visual self-distillation. iBOT’s 2021 preprint, published at ICLR 2022, adds an online masked-patch target. DINOv2 (2023) scales global and local learning with data curation, stabilization and efficient execution.

LeJEPA (2025) is a parallel branch: shared-encoder agreement with explicit distribution regularization. TIPSv2 (2026) combines vision-language supervision with all-token patch distillation and a lighter teacher path. It is not simply the next version of LeJEPA.

Use chronology to track motivating failures: local targets, scale, collapse control, dense text alignment. These branches can share useful ideas without sharing every target, normalization or gradient contract.

Sources and connected topics

• DINO (2021) ↗ · global visual self-distillation.
• iBOT (2021 preprint/2022 conference) ↗ · online patch targets, shared heads and evaluation.
• DINOv2 (2023) ↗ · global/local scale, KoLeo and crop execution.
• LeJEPA (2025) ↗ · centered agreement, SIGReg and conditional theory.
• TIPSv2 (2026) ↗ · TIPSv2 project ↗ · iBOT++, head-only EMA and caption granularity.

Use Losses and Objectives for general loss gradients, Probability and Distributions for distribution contracts, and Distillation for frozen/EMA/on-policy teacher distinctions. For implementation work, preserve shapes, mask ownership and reductions before tuning a loss weight.