Riemannian Geometry for Pre-trained Language Model Embeddings

2026-07-09 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

A new method, Riemannian Mean Pooling (RMP), investigates whether sentence-level classification signals reside in the Riemannian geometry of contextual token embeddings. RMP extracts per-token pullback metrics from a learned encoder's analytical Jacobian and aggregates them using the Fréchet mean on the symmetric positive definite (SPD) manifold. Evaluated across CoLA, CREAK, and RTE datasets, RMP consistently outperformed Euclidean mean pooling, achieving 0.778 AUC on CoLA, 0.703 AUC on CREAK, and 0.635 AUC on RTE. On FEVER-Symmetric, a negative control dataset, RMP correctly performed at chance (0.508 AUC). Ablation studies revealed that the geometric aggregation, rather than the trained encoder, is the primary source of performance gains on CoLA and RTE, with the trained encoder contributing additional signal specifically on the knowledge-heavy CREAK dataset.

Key takeaway

For NLP Engineers developing sentence representation models, this research suggests that incorporating Riemannian geometry can significantly enhance classification performance beyond traditional Euclidean pooling. You should consider implementing Riemannian Mean Pooling (RMP) to leverage the local geometric structure of token embeddings, especially for tasks requiring nuanced linguistic understanding. This approach offers a robust alternative to standard aggregation methods, even with randomly initialized encoders, indicating a fundamental improvement in how sentence-level signals are captured.

Key insights

Riemannian aggregation of token embeddings improves sentence-level classification by capturing local geometric structure.

Principles

• Geometric aggregation can outperform Euclidean pooling.
• Local manifold-like structure carries classification signal.
• Encoder training is less critical than geometric aggregation for some tasks.

Method

Riemannian Mean Pooling (RMP) extracts per-token pullback metrics via an encoder's Jacobian, aggregates them using the Fréchet mean on the SPD manifold, and classifies in the tangent space.

In practice

• Use RMP for improved sentence-level classification.
• Apply Fréchet mean for geometric aggregation on SPD manifolds.

Topics

• Riemannian Geometry
• Language Model Embeddings
• Sentence Representation
• Fréchet Mean
• Pullback Metrics
• NLP Classification

Code references

• hotherio/intrinsic-green-learning

Best for: AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Riemannian Geometry for Pre-trained Language Model Embeddings — Sentence-level classification signals can be effectively extracted from pre-trained language model embeddings using Riemannian geometry.
• Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing — Controlled semantic variation induces measurable, nonlinear local geometry in sentence embedding spaces.
• Riemannian Geometry for Pre-trained Language Model Embeddings — The geometric structure of PLM embeddings, particularly via Riemannian Mean Pooling, enhances sentence-level classification signal.
• Geometry of Semantic Space: Comparative Study of Discrete and Continuous Models — Graph-based semantic models offer more interpretable and structured meaning representations than high-dimensional transformer embeddings.
• [NLP Series] How Machines Turn Words into Numbers: Embeddings — Embeddings turn language into geometry, allowing machines to grasp semantic relationships.
• Gating Enables Curvature: A Geometric Expressivity Gap in Attention — Multiplicative gating in attention enables non-flat geometric representations, enhancing expressivity for nonlinear tasks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.