Cross-model Parallelism
Accepted at NeurIPS 2026
Displacement GeometryCaptures Platonic Shared RealityAcross Models and Modalities
Department of Computer Science, Dartmouth College
* chenming.shang@dartmouth.edu
TL;DR: We show that what models share is not the location of samples in representation space, but the directions (displacement vectors) between them. Under orthogonal alignment, these vectors are substantially preserved across independently trained vision and language encoders.
Two maps of the same city need not agree on coordinates to give you similar directions. The displacement vectors between landmarks are preserved, even when their absolute positions on the page are not.
Neural networks enjoy an analogous freedom: each learns its own coordinate system. After alignment, does the displacement between two samples in one model point in the same direction as the displacement in another?
Geometry
Within a model, parent concepts are approximately orthogonal to their child variation vectors; across models, concept displacements are parallel.
Within-model Orthogonality
Experiments
We evaluate 44 pretrained models: 25 vision transformers spanning three sizes and six pretraining paradigms, plus 19 language models. Cross-model comparisons use rigid alignment only.
Parallelism
Concept-level parallelism exceeds sample-level parallelism. Under rigid alignment alone, displacement vectors between concepts are preserved across independently trained models, while their pointwise positions differ. Cross-Model similarity increases with model size and is near zero for randomly initialized networks.
Sample-Level Agreement
Concept-Level Agreement
Effect Of Model Scale
All panels report cosine similarity. PCA, whitening, and Procrustes alignment are fitted only on the alignment split and kept fixed during evaluation on disjoint inputs. After alignment, each model’s mean, projected through its own PCA and whitening transforms, is restored. This shift affects Pointwise and Cross-Object similarity but cancels in Cross-Model displacements.
Orthogonality
Parent concepts remain approximately orthogonal to their child variation vectors across layers. If samples are weighted combinations of concepts, we hypothesize that within-model orthogonality and cross-model concept parallelism explain sample-level parallelism. Degrading either concept-level condition weakens sample-level parallelism in controlled simulations.
Geometry Across Layers
Disrupting Concept Parallelism
Disrupting Orthogonality
Modality
The pattern of Cross-Model similarity differs systematically by modality. Vision-model agreement concentrates at corresponding layers; language-model agreement extends broadly across middle and deep layers. Nearly every language-model layer aligns with deep vision-model layers. This cross-modal pattern is consistent with shared semantic structure.
Vision ↔ Vision
Language ↔ Language
Vision ↔ Language
Dynamics
Pointwise and Cross-Model similarity respond differently to the training objective. From pretrained models, fine-tuning leaves Pointwise nearly unchanged while reducing Cross-Model; feature distillation raises both; relational distillation raises Cross-Model while leaving Pointwise nearly unchanged. From random initialization, supervised training increases both similarities, relational distillation further improves Cross-Model alignment, and feature distillation yields larger gains in both.
Fine-Tuning
Relational Distillation
Feature Distillation
Training From Scratch
Scratch + Relational Distillation
Scratch + Feature Distillation
Decomposition
The shared component aligns linearly across models but remains nonlinearly entangled with the private component. At matched dimensions, it has higher Cross-Model similarity and retains more linearly decodable semantic content. Linear dependence between the components is low; RBF dependence is substantially higher.
Cross-Model Agreement
Linearly Decodable Semantics
Nonlinear Dependence
Application
ShadowCLIP
ShadowCLIP instantiates Shadow Casting: capability transfer via a cached forward pass. A frozen CLIP backbone is augmented by two lightweight residual modules, using DINO representations and relational distillation from an LLM teacher.
Training
Cached Representations
Freeze the CLIP, DINO, and LLM backbones and cache their representations once over the training set.
Vision and Language Residuals
A SwiGLU residual uses CLIP as a gating signal and DINO as a value; a two-layer MLP augments CLIP text features.
Training Objective
Train on CC12M with image–text contrastive loss and relational distillation from an LLM teacher. Zero-initialized gates make ShadowCLIP exactly CLIP at initialization.
Inference
At inference, the LLM teacher is no longer needed, while the vision branch still uses DINO to encode unseen images. ShadowCLIP retains the CLIP text encoder and its trained residual for inference efficiency.
Compared with full fine-tuning of CLIP-Large, ShadowCLIP requires 95% fewer trainable parameters (20M vs. 400M). Caching reduces training-phase FLOPs by a factor of 2,000; including the one-time precomputation, total FLOPs over three training epochs are 9× lower.
Experimental Results
We evaluate on zero-shot classification (11 benchmarks), image–text retrieval (MSCOCO, Flickr30k), and visual-centric understanding (MMVP).
| Method | ImageNet-1K | COCO I2T | COCO T2I | Flickr30k I2T | Flickr30k T2I | MMVP |
|---|---|---|---|---|---|---|
| CLIP | 72.7 | 59.7 | 43.0 | 87.6 | 70.2 | 20.0 |
| SAIL-NV2 | 72.1 | 57.3 | 45.3 | 84.9 | 73.0 | 28.0 |
| CLIP-FT | 69.3 | 55.6 | 42.7 | 82.5 | 68.9 | 19.3 |
| w/o LG | 70.2 | 56.6 | 43.5 | 82.1 | 70.3 | 20.0 |
| ShadowCLIP | 76.5 | 64.0 | 44.4 | 89.9 | 72.4 | 29.6 |
LG denotes the Language Guider. The w/o LG variant removes the language-side MLP residual and relational distillation, fine-tuning only the language-side linear projection. I2T / T2I: image-to-text / text-to-image retrieval R@1. Bold marks the best result within the selected model size; “—” indicates an unreported result.
SAIL uses an LLM text encoder at inference; ShadowCLIP retains CLIP’s text encoder. Computational figures above concern training, and include caching only where specified.
Citation
If you find this work useful, please consider citing our paper.
@article{shang2026displacement,
title = {Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities},
author = {Shang, Chenming and Tang, Yujin and Ouyang, Junjie and
Xu, Ruize and Breuer, Adam and Singh, Nikhil},
journal = {arXiv preprint arXiv:2609.24209},
year = {2026},
eprint = {2609.24209},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arXiv.2609.24209},
url = {https://arxiv.org/abs/2609.24209}
}