The Practical Side of Representation Geometry
Deciding which pretrained models to join, and how, before the training run

When a finding about representations becomes practical

The Platonic Representation Hypothesis argues that as networks scale, their representations converge to a shared model of the world, a trend reported across the usual representational similarity metrics. Our earlier work, the Aristotelian Representation Hypothesis [3], re-examines that claim. After correcting for two confounders that these similarity metrics have, the trend reported by global metrics largely flattens, while the trend from local neighborhood metrics holds. What reliably converges is local neighborhood structure: which points are each other's nearest neighbors.

Local neighborhood alignment is typically measured by mutual $k$-NN: embed a small probe set through each model and, for every sample, take the fraction of its $k$ nearest neighbors that both representation spaces share. This needs no training, and the cost grows with the number of models, not the number of pairs: each model is embedded once, and its neighbors are reused against every candidate partner.

Here we show how to make these convergence findings practical, by connecting what two models share locally to how you combine them. COMPASS predicts, before training, which pairs are worth combining. PORTAL uses the same understanding to replace the training of a projector with a closed-form construction.

Whether to connect them COMPASS · NeurIPS 2026

Combining two pretrained models can mean multiple things. You can align them, learning a map that lines up their two representation spaces so the same inputs land in the same place, as when a vision encoder and a text encoder are tied together for retrieval (cross-modal alignment). Or you can merge two models fine-tuned on different tasks into a single set of weights (task arithmetic). Either way, whether a given pair combines well is normally known only after you run that alignment or merge, so choosing a pair means a search over training runs. COMPASS [1] gives the answer first, as a score computed from forward passes.

COMPASS scores a pair on two numbers, both built from the same local overlap: for each sample in a small probe set, the fraction of its $k$ nearest neighbors that the two models share.

• Ease is pair-specific. How much does this pair already agree, before any training (before encoder training in alignment, before merging in task arithmetic)? Two models that already agree on which points are nearest neighbors need little training to align fully. Two that start far apart need much more.
• Ceiling is paradigm-level. How much agreement can the two training paradigms support at all, averaged over models drawn from them? Paradigms with very different objectives, say a reconstructive encoder and an autoregressive language model, can only agree so far, however well a single pair happens to start.

The widget below runs step 1. Watch the local neighborhood overlap, and the ease and ceiling behind it, change as you move the two sliders.

The four regimes read off the two axes. Act-now pairs (high ease, high ceiling) are the strongest immediate candidates. Budget-limited pairs (low ease, high ceiling) can still succeed, but need more data, more optimization, or a stronger partner. Ceiling-limited pairs (high ease, low ceiling) already agree, but their paradigms limit the agreement they can reach, so a paradigm whose objective is closer to the target helps more. Incompatible pairs are low on both axes and are deprioritized.

How to connect them PORTAL · NeurIPS 2026

A multimodal LLM is built in two stages. First a projector, a small module, is trained on hundreds of thousands of image-caption pairs to map visual tokens into the language model's input space (LLaVA and the models after it). Then the full model is trained together. This first stage is treated as essential, yet what it learns has stayed unclear, and earlier reports even disagree on whether it helps. PORTAL [2] makes a concrete claim: the stage performs only a coarse alignment, moving visual features into a region the language model can read, which many different projectors do equally well.

Two findings say the stage learns only a coarse alignment. First, the interpolation in the middle panel: if training reached one precise, unique projector, averaging two independently trained projectors would break it, yet the average holds and often beats both. Second, only $10$ to $20\%$ of the standard $558$k-pair corpus already recovers most of the downstream gain. Neither a broad interpolation path nor such early saturation fits a narrow target.

Since many projectors work, the stage may not need gradient descent to find one. PORTAL (Projector via Optimal-tRansport ALignment) initializes the projector in closed form from each modality's statistics. It keeps the standard two-layer MLP projector but fixes the second layer to the identity, so only the first affine map $(\mathbf{W}_1, \mathbf{b}_1)$ has to be chosen, as the Gaussian optimal-transport map between the vision and language feature distributions:

Here $\boldsymbol{\mu}_s, \boldsymbol{\mu}_t$ are the means, $\mathbf{V}_s, \mathbf{V}_t$ the top-$k$ principal directions, and $\boldsymbol{\sigma}_s, \boldsymbol{\sigma}_t$ the spread along each, for the vision (source) and language (target) sides. The map whitens the vision features, rotates them into the language model's principal directions, and rescales each to the language model's spread, and $\mathbf{b}_1$ recenters on its mean. It needs one PCA per side, no paired captions, and no gradient steps, and it removes the projector training stage rather than shortening it. The same construction also initializes other projector designs, a plain linear projector and the larger C-Abstractor, not only the two-layer MLP.

Across seven (vision encoder, language model) pairs spanning four language-model families (Vicuna, LLaMA, Qwen, OLMo) from 0.5B to 8B parameters and two vision encoders (CLIP, SigLIP), the training-free projector matches the standard pipeline trained on $558$k image-caption pairs, and beats it in three of the seven settings, peaking at $106\%$ of the trained baseline. The gap that remains is on OCR-heavy benchmarks, where fine character-level detail still rewards the paired supervision that per-modality statistics cannot capture.

The common idea

COMPASS and PORTAL use the same idea twice. COMPASS compares the local neighborhoods of two models to predict, before training, which pairs are worth combining. PORTAL uses the feature geometry of the vision and language sides to build the projector that links them in closed form. In both, understanding the structure that representations share does practical work: it lets COMPASS rank pairs before training, and lets PORTAL build a projector without it.