Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference

2026-06-15 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

A new approach, Orthogonally Equivariant Transformers, specifically ConjFormer, addresses privacy concerns in large language model (LLM) inference when sensitive text is processed by third-party providers. Current split inference methods, which send only hidden representations, are vulnerable to token recovery via nearest-neighbor search against public embedding tables. The proposed orthogonal obfuscation procedure involves the client multiplying embeddings by a secret orthogonal matrix before transmission. ConjFormer, an O(d)-equivariant transformer variant, enables correct inference under these arbitrary rotations through scalar RMSNorm and blockwise orthogonal conjugation of linear weights. Experiments on GPT-2 and Llama 3.2 1B models fine-tuned on PubMed demonstrate that this method eliminates direct cosine nearest-neighbor inversion and reduces token recovery from over 35% top-10 to at most 1.3%, with only a 0.4% perplexity increase.

Key takeaway

For Machine Learning Engineers deploying LLMs with split inference on sensitive data, this research offers a robust privacy defense. You should consider integrating orthogonal obfuscation and ConjFormer's O(d)-equivariant architecture to drastically reduce token recovery risks from hidden representations. This approach achieves strong privacy with minimal perplexity impact, making it a practical alternative to heavy cryptographic methods.

Key insights

Enforcing architectural symmetry via orthogonal obfuscation and equivariant transformers provides practical privacy for LLM inference.

Principles

• Hidden representations are vulnerable to nearest-neighbor token recovery.
• Orthogonal obfuscation can eliminate direct cosine nearest-neighbor inversion.
• Architectural symmetry offers a practical privacy defense without noise.

Method

The client applies a secret orthogonal matrix to embeddings; the server uses ConjFormer, an O(d)-equivariant transformer with scalar RMSNorm and blockwise orthogonal conjugation, to process in the rotated basis.

In practice

• Apply client-side orthogonal obfuscation to embeddings.
• Implement ConjFormer for O(d)-equivariant LLM inference.
• Fine-tune existing LLMs (e.g., GPT-2, Llama 3.2 1B) with ConjFormer.

Topics

• LLM Privacy
• Orthogonal Equivariance
• ConjFormer
• Split Inference
• Data Obfuscation
• Machine Learning Security

Best for: AI Architect, Research Scientist, CTO, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Separable Expert Architecture: Toward Privacy-Preserving LLM Personalization via Composable Adapters and Deletable User Proxies — A three-layer architecture enables privacy-preserving LLM personalization through composable adapters and deletable user proxies.
• Communication-Efficient Verifiable Attention for LLM Inference — VeriAttn accelerates verifiable LLM inference by offloading attention computation to GPU for TEE verification.
• Privacy-Preserving LLMs Routing — PPRoute enables privacy-preserving LLM routing via MPC-friendly operations, multi-stage training, and an efficient unsorted Top-k algorithm.
• RAG Security and Privacy: Formalizing the Threat Model and Attack Surface — RAG systems introduce unique privacy and security risks requiring a formal threat model and specific defense strategies.
• Differential Privacy of Gaussian Process Posterior Sampling — Intrinsic randomness in GP posterior sampling provides differential privacy, critically dependent on effective ridge regularization.
• Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation — Censored Chinese LLMs serve as a natural testbed for evaluating honesty elicitation and lie detection techniques.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.