arxiv:2601.09173

Geometric Stability: The Missing Axis of Representations

Published on Oct 1

· Submitted by

Prashant Raju on Jan 15

Upvote

Authors:

Prashant C. Raju

Abstract

Geometric stability measures representational robustness under perturbation, offering complementary insights to similarity metrics in analyzing learned representations across diverse domains.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Representational similarity methods compare the geometries of neural representations, but they do not measure how consistently the geometry of a single representation is recovered from subsets of its feature coordinates. We call this property geometric stability and introduce Shesha, which estimates it by correlating representational dissimilarity matrices from complementary random feature subsets. Shesha is not invariant to orthogonal rotations: representations with identical Gram matrices, and therefore identical linear CKA, can have different geometric stability. Controlled transformations further separate the quantities. Across 2{,}463 encoder configurations spanning seven domains, similarity and stability are positively associated across non-PCA transformations (ρ=+0.75) but negatively associated under PCA-coordinate compression (ρ=-0.47). We further evaluate 170 pretrained vision models across six datasets. DINOv2 combines strong transfer performance with bottom-quartile stability on five of six datasets, showing that transferability and feature-split stability need not coincide. Across random feature subsets, the marginal relationship between Shesha and linear-probe variability is dataset-dependent; after controlling for task alignment with LogME, higher Shesha is associated with lower variability on five of six datasets. These results identify geometric stability as a basis-dependent property that complements representational similarity and task alignment.

View arXiv page View PDF GitHub 3 auto Add to collection

Community

pcr2120

Paper author Paper submitter Jan 15

•

edited Jan 15

DeepSeek got it half right with their mHC paper: stability matters for scaling. But they only measure stability DURING training.

What about the stability of what models LEARN?

I built Shesha to measure this - a geometric stability metric with SOTA results across AI Safety, Constitutional AI, Model selection, and CRISPR perturbation analysis.

The core insight: Most evals check external similarity (does output X match Y?). But imagine a massive library where someone reshuffled all the books. A content-based audit would say that nothing's wrong since the inventory is identical. But the library is useless since nothing can be found. That's the gap Shesha fills.

The implications are broad with SOTA results across 4 domains:

AI Safety - Shesha is the best canary in the coal mine. Shesha outperforms CKA and Procrustes on drift detection. Detects 2x more drift than CKA, triggers earlier 73% of the time, catches subtle LoRA shifts at 90% sensitivity (5% FPR) - with only 7% false alarms vs Procrustes' 44%.
Constitutional AI - Shesha provides the best steering prediction. Constitutional AI needs models you can actually steer. Most metrics ask: "Are classes separable?" Wrong question! Shesha asks: "Is that separation STABLE under perturbation?" Tested on 35-69 embedding models across 3 experiments. Shesha outperforms Fisher discriminant, silhouette score, Procrustes, and anisotropy. The correlations with intervention success are rho=0.89-0.96, and the partial correlations after controlling for separability are rho=0.67-0.76. Stability ≠ separability.
Model selection - Shesha exposes what LogME misses. The DINO Paradox - best transfer scores, worst geometric stability. Tested 94 vision models on 6 datasets: DINOv2 ranked #1 in LogME on 4/6 datasets but last or near last in stability on 5/6. SOTA transfer incurs a "geometric tax."
CRISPR perturbations - Shesha serves as a new filter for target selection. CRISPR screens find hits by magnitude, but magnitude alone can't distinguish clean lineage drivers from promiscuous regulators. Shesha adds precision. Tested on 811 perturbations, Shesha showed uniformly positive magnitude-stability correlations ranging from rho=0.746 in high-variance screens to rho=0.963 in cleaner activation settings. Notably, in discordant cases, Shesha separates KLF1 (stable, specific) from CEBPA (strong but messy) purely from geometry.

Additional validation in neuroscience: Geometric stability predicted neural-behavioral coupling (rho=0.18, p=0.005) in Neuropixels data. Centroid drift showed no relationship (rho=0.00). Stability ≠ consistency.

Try it yourself:
PyPI: pip install shesha-geometry
Tutorials: https://github.com/prashantcraju/shesha?tab=readme-ov-file#tutorials

Preprint: https://arxiv.org/abs/2601.09173
Code: https://github.com/prashantcraju/geometric-stability