RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

RoME, a novel Robust Mixture of Low-Rank Experts model, addresses the robustness trade-offs inherent in multi-perturbation adversarial training (MAT) against diverse ℓ_p perturbations. Existing Mixture of Experts (MoE) approaches struggle with experts overlooking threat-specific features and redundant feature capture, alongside threat-agnostic gating that prevents distinct model pathways. RoME tackles this by implementing each expert as a low-rank additive update to a shared backbone, enabling experts to focus on threat-specific information while the backbone handles common features. Furthermore, RoME introduces dual-scale gating, which leverages local and global features for threat discrimination, and threat-guided gating diversification to ensure varied expert utilization across different threats. Extensive experiments show RoME surpasses current state-of-the-art MAT in union robustness and natural accuracy, also enhancing robustness against previously unseen threats. Its code is available on GitHub.

Key takeaway

For AI Security Engineers developing robust models against diverse adversarial attacks, RoME offers a promising architectural approach. If your current multi-perturbation adversarial training (MAT) struggles with robustness trade-offs or threat-agnostic expert routing, consider integrating low-rank additive expert updates and dual-scale gating. This method can significantly improve union robustness and natural accuracy, even against unseen threats, by creating threat-specific model pathways.

Key insights

RoME uses low-rank experts and diversified gating to achieve robust adversarial training against multiple, diverse threats.

Principles

• Experts should capture threat-specific information.
• Shared backbone can handle threat-common features.
• Gating networks need threat-discriminative signals.

Method

RoME employs low-rank additive expert updates on a shared backbone. It uses dual-scale gating with local/global features and threat-guided diversification for expert routing.

In practice

• Implement low-rank expert updates for efficiency.
• Use dual-scale gating for threat discrimination.
• Enforce diverse expert utilization across threats.

Topics

• Adversarial Robustness
• Mixture of Experts
• Low-Rank Adaptation
• Multi-perturbation Adversarial Training
• Computer Vision
• Deep Learning Architectures

Code references

• wkim97/RoME

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• GuardNet: Ensemble Strategies of Shallow Neural Networks for Robust Prompt Injection and Jailbreak Detection — Robustness against LLM adversarial attacks may depend more on diverse example coverage and threshold calibration than on model scale.
• GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. — Mixture of Experts (MoE) enables large models to activate only a fraction of parameters per token, boosting efficiency.
• Continuously hardening ChatGPT Atlas against prompt injection — Automated red teaming with reinforcement learning proactively hardens AI agents against prompt injection attacks.
• Harnessing non-adversarial robustness in large language models — LLM robustness to prompt variations can be efficiently acquired via fine-tuning, avoiding full retraining.
• Beyond Uniform Sampling: Synergistic Active Learning and Input Denoising for Robust Neural Operators — Combining active learning with input denoising synergistically improves neural operator robustness against adversarial attacks.
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal — Transforming PLM embedding spaces to approximate Gaussian properties enhances adversarial robustness without costly adversarial training.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.