PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry

2026-06-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

PermDoRA investigates adapter interference in large language models (LLMs), challenging the common hypothesis that interference arises from overlap in linear parameter updates. The study utilized DoRA-RBAC, a hierarchical adapter composition framework, comparing conventional Euclidean merging with a geometry-aware Riemannian-inspired strategy that approximates the Frechet mean via normalized directional averaging. This was tested across multiple QA benchmarks (GPQA, PubMedQA, SimpleQA, WMDP) using LLaMA-3.1-8B and Mistral-7B. Results indicate that while single-domain performance matches LoRA, geometry-aware merging offers no consistent advantage over standard averaging in multi-domain settings. Diagnostic analysis further shows that angular alignment and orthogonality of adapter updates are weak predictors of composition performance, suggesting interference is consistent with interactions in shared nonlinear representations.

Key takeaway

For AI scientists designing modular access control or multi-domain adapter systems for LLMs, this research indicates that focusing solely on parameter-space geometry for interference mitigation may be unproductive. Your efforts should instead investigate interactions within shared nonlinear representations. Consider standard averaging for multi-domain adapter merging, as geometry-aware methods showed no consistent advantage over simpler approaches.

Key insights

Adapter interference in LLMs is not primarily governed by parameter-space geometry.

Principles

• Adapter interference aligns with shared nonlinear representation interactions.
• Angular alignment and orthogonality are weak predictors of composition performance.

Method

The DoRA-RBAC framework compares conventional Euclidean merging with a Riemannian-inspired strategy, approximating the Frechet mean via normalized directional averaging.

In practice

• Single-domain adapter performance matches LoRA.
• Standard averaging is as effective as geometry-aware merging for multi-domain settings.

Topics

• Adapter Interference
• Large Language Models
• DoRA-RBAC
• Parameter-Space Geometry
• Multi-domain LLMs
• Nonlinear Representations

Best for: Research Scientist, AI Scientist

Related on AIssential

• From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models — Integrating geometric spatial representations directly into LLMs enables robust, true spatial reasoning beyond symbolic pattern matching.
• Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses — LLMs inconsistently adjust response complexity, failing to reliably differentiate jargon and information across user-controlled levels.
• The Eigenvectors of AI: Shared LoRA Subspaces for Continual Learning — Task-specific LoRA adapters converge to a shared low-rank subspace, enabling massive storage and memory reduction.
• Weight Patching: Toward Source-Level Mechanistic Localization in LLMs — Weight Patching identifies where LLM capabilities are encoded in parameters, distinguishing source carriers from aggregation modules.
• Mixture-of-Depths Attention — MoDA improves LLM depth scaling by allowing attention heads to access KV pairs from multiple layers, mitigating signal degradation.
• A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM$Δ$ Integration into Upcycled MoE — Upcycling dense LLMs into MoE architectures with parameter delta integration efficiently expands language capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.