Transformer Geometry Observatory TGO-II: Representational Similarity Observatory

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

Transformer Geometry Observatory-II (TGO-II) is a new analysis framework designed to investigate the geometric evolution of Transformer representations during supervised training. Focusing on Vision Transformer (ViT-Small/16) models, TGO-II employs Centered Kernel Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA), Two-Nearest Neighbor Intrinsic Dimensionality (TwoNN-ID), and token covariance analysis. Experiments using TGO-II revealed three key observations: CKA and SVCCA progressively decrease during training, indicating increasing representational specialization across layers. Intrinsic dimensionality consistently increases before stabilizing, suggesting a progressive expansion of the representation manifold. Finally, token covariance and coupling analyses show strong token interaction structure persists throughout training, challenging the idea that complexity arises from token independence. These findings suggest representation complexity and layer specialization emerge simultaneously, with manifold expansion occurring without token decoupling.

Key takeaway

For AI Scientists and Research Scientists investigating Vision Transformer behavior, understanding that representational complexity emerges alongside layer specialization without token decoupling is crucial. Your architectural decisions and training strategies should account for the persistent strong token interaction structure, rather than assuming increasing independence. Consider TGO-II's analytical methods to gain deeper insights into how your models' internal representations evolve, potentially guiding more effective model design and optimization.

Key insights

Transformer representations specialize and expand their manifold while maintaining strong token interactions during training.

Principles

• Representational specialization increases with training.
• Intrinsic dimensionality expands then stabilizes.
• Strong token interaction persists throughout training.

Method

TGO-II analyzes Vision Transformer representations using CKA, SVCCA, TwoNN-ID, and token covariance to track geometric evolution during supervised training.

In practice

• Use CKA/SVCCA to track layer specialization.
• Monitor intrinsic dimensionality for manifold expansion.
• Analyze token covariance for interaction structure.

Topics

• Vision Transformers
• Representation Geometry
• Centered Kernel Alignment
• Singular Vector Canonical Correlation Analysis
• Intrinsic Dimensionality
• Token Covariance Analysis

Best for: AI Scientist, Research Scientist, Computer Vision Engineer

Related on AIssential

• Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory — Vision Transformer training redistributes representational variance across dimensions, increasing effective dimensionality rather than concentrating information.
• Tracking Representation Dynamics in Large Language Models with Persistent Homology — Persistent homology uncovers internal representation dynamics in LLMs during fine-tuning, showing early topological reorganization and objective-specific trajectories.
• Layerwise Dynamics for In-Context Classification in Transformers — Enforcing task symmetries reveals a closed-form, layerwise algorithmic recursion in transformers for in-context classification.
• Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap — MoE experts show functional decorrelation but overlapping representations, modulated by routing sparsity.
• Why Transformer Representations Tend to Live on Hyperspheres — Transformer representations cluster on hyperspheres due to normalization and high-dimensional geometry, making direction paramount.
• Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective — Neural network generalization is geometrically characterized by the Representation Gap, scaling with intrinsic data dimension and model equivariance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.