RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

RayRoPE is a novel positional encoding scheme for multi-view transformers, developed by researchers from Carnegie Mellon University and Apple. It addresses limitations in prior encoding methods by uniquely representing image patches, allowing SE(3)-invariant attention with multi-frequency similarity, and adapting to underlying scene geometry. RayRoPE encodes patch positions using associated rays and a predicted 3D point along the ray, rather than just direction. It computes query-frame projective coordinates for multi-frequency similarity and analytically computes expected position encoding under depth uncertainty. Validated on novel-view synthesis and stereo depth estimation, RayRoPE consistently improves performance, showing a 15% relative improvement on LPIPS in CO3D and seamlessly incorporating RGB-D input for larger gains.

Key takeaway

For Machine Learning Engineers developing 3D vision applications like novel-view synthesis or stereo depth estimation, adopting RayRoPE can significantly improve model performance and geometric consistency. You should consider integrating its ray-based, geometry-adaptive positional encoding, especially when dealing with diverse camera poses or uncertain depth information. This approach offers superior high-frequency detail and 3D consistency compared to traditional methods, making your models more robust and accurate.

Key insights

RayRoPE enhances multi-view transformers with geometry-adaptive, SE(3)-invariant, multi-frequency positional encoding using predicted ray depths.

Principles

• Positional encodings must be SE(3)-invariant.
• Geometry-adaptiveness improves multi-view attention.
• Uncertainty modeling stabilizes depth predictions.

Method

RayRoPE defines patch positions as ray segments (camera center, predicted 3D point at depth d), projects them to the query camera frame, and applies Rotary Positional Embedding (RoPE) with multi-frequency similarity. It models depth uncertainty via expected RoPE.

In practice

• Integrate predicted depth and uncertainty into multi-view transformers.
• Apply expected RoPE for robust encoding under depth ambiguity.
• Incorporate known RGB-D input for enhanced geometric accuracy.

Topics

• Positional Encoding
• Multi-view Transformers
• Novel-View Synthesis
• Stereo Depth Estimation
• Rotary Positional Embedding
• SE(3) Invariance

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation — RayPE integrates 3D camera ray geometry into video diffusion transformer self-attention for enhanced consistency.
• URoPE: Universal Relative Position Embedding across Geometric Spaces — URoPE extends RoPE to cross-view and cross-dimensional geometric spaces via explicit projective geometry and depth anchors.
• Understanding Positional Embeddings in Transformers (with Intuition and Examples) — Positional embeddings are crucial for Transformers to understand sequence order, evolving from additive to rotation-based methods.
• Robustness of Similarity-based Positional Encoding Under Rotations: Theoretical Analysis and Experimental Validation — Similarity-based positional encoding (simPE) provides stable performance under rotational perturbations in Transformers, outperforming standard learned methods.
• Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention — Positional schemes fundamentally determine the spectral algebra and functional behavior of attention heads in pretrained models.
• Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity — RoPE enhances Transformer expressivity by reducing functional equivalence symmetry compared to sinusoidal encodings.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.