RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

Robust Mixture of Low-Rank Experts (RoME) is a novel framework designed to enhance multi-perturbation adversarial training (MAT) by mitigating robustness trade-offs across diverse ℓ₊ threats. Traditional Mixture of Experts (MoE) approaches struggle with threat-agnostic routing and redundant feature learning. RoME addresses this by implementing each expert as a low-rank additive update to a shared backbone, enabling experts to capture threat-specific information while the backbone handles common features. It further introduces dual-scale gating, which leverages both local patch-level and global image-level features for improved threat discrimination, and threat-guided gating diversification to ensure distinct expert utilization across threats. Extensive experiments on CIFAR-10, ImageNet-100, and ImageNet-1K demonstrate RoME's superior performance, achieving leading union robustness and natural accuracy, and improving resilience against unseen threats, all while maintaining computational efficiency with only 1.04x more parameters and 1.17x more training time.

Key takeaway

For Machine Learning Engineers developing robust models against diverse adversarial threats, RoME offers a computationally efficient solution to overcome robustness trade-offs. You should consider integrating its low-rank expert architecture with dual-scale gating and diversification techniques to create threat-specific model pathways. This approach significantly improves union robustness and natural accuracy, even against unseen attacks, without substantial increases in model parameters or training time, demonstrating performance that surpasses many existing methods.

Key insights

Multi-perturbation adversarial training benefits from threat-specific model pathways via low-rank experts and diversified dual-scale gating.

Principles

• Adversarial threats induce distinct distributional shifts.
• Shared backbones capture threat-common features.
• Dual-scale features improve threat discrimination.

Method

RoME employs low-rank experts as additive updates to a shared backbone. It uses dual-scale gating (local/global features) and threat-guided gating diversification to enforce distinct expert routing for different adversarial threats.

In practice

• Apply low-rank experts for threat-specific learning.
• Combine local and global features for gating.
• Use regularization to diversify expert routing.

Topics

• Multi-perturbation Adversarial Training
• Mixture of Experts
• Low-Rank Adaptation
• Adversarial Robustness
• Threat-Agnostic Routing
• Dual-Scale Gating
• Vision Transformers

Code references

• wkim97/RoME
• microsoft/LoRA

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations — RoME uses low-rank experts and diversified gating to achieve robust adversarial training against multiple, diverse threats.
• Harnessing non-adversarial robustness in large language models — LLM robustness to prompt variations can be efficiently acquired via fine-tuning, avoiding full retraining.
• Using Topological Data Analysis to Characterize the Layers of Language Models Before and After Word Substitution Attacks — Topological Data Analysis reveals systematic structural changes in LLM internal representations due to adversarial word substitutions.
• GuardNet: Ensemble Strategies of Shallow Neural Networks for Robust Prompt Injection and Jailbreak Detection — Robust LLM prompt injection and jailbreak detection prioritizes diverse examples and threshold calibration over model scale.
• On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces — Adversarial vulnerability in vision-language models stems from the spectral structure of intermediate linear transformations.
• Attacking the Spike: On the Transferability and Security of Spiking Neural Networks to Adversarial Examples — Adversarial attacks on SNNs are highly dependent on surrogate gradient estimators, and a new method, MDSE, exploits this for cross-model effectiveness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.