Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO

2026-06-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

AdvGRPO introduces a co-training framework that makes the GRPO reinforcement learning algorithm viable for joint attacker-defender optimization in language model red teaming, addressing previous instability issues. This method utilizes dense multi-channel rewards and decoupled advantage normalization to stabilize GRPO. Training progresses through a curriculum, starting with single-turn attacks, moving to closed-loop multi-turn attacks, and then bootstrapping co-training where attacker and defender models are updated alternately. The framework successfully produces highly effective and transferable attacks, and the resulting co-trained defenders demonstrate superior performance on established safety benchmarks compared to baseline models.

Key takeaway

For AI Security Engineers focused on developing adaptive red teaming strategies and robust LLM defenses, AdvGRPO presents a viable reinforcement learning co-training framework. You should consider integrating this GRPO-based approach to discover novel, transferable attacks and significantly enhance your language models' resilience against evolving threats. This method offers a structured way to improve safety benchmark performance.

Key insights

AdvGRPO enables stable GRPO-based co-training for adaptive LLM red teaming, yielding effective attacks and robust defenders.

Principles

• Adaptive red teaming requires evolving attacker-defender co-training.
• Dense multi-channel rewards improve GRPO stability.
• Curriculum learning enhances co-training effectiveness.

Method

AdvGRPO uses dense multi-channel rewards and decoupled advantage normalization for GRPO stability. Training follows a curriculum from single-turn to multi-turn attacks, then bootstraps alternating attacker-defender co-training.

In practice

• Apply AdvGRPO for robust LLM safety benchmark improvements.
• Utilize curriculum training for complex attack generation.
• Implement decoupled advantage normalization for GRPO stability.

Topics

• AI Red Teaming
• Reinforcement Learning
• GRPO
• Language Models
• Attacker-Defender Co-training
• Safety Benchmarks

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Security Engineer

Related on AIssential

• AI Red Teams for Adversarial Training: How to Make ChatGPT and LLMs Adversarially Robust — AI Red Teams are crucial for identifying and mitigating adversarial vulnerabilities in large language models.
• AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO — Adaptive GRPO enhances text-to-image flow models by dynamically matching training prompts to model capability and fusing local-global advantage estimates.
• GRPO Does Not Close the Multi-Agent Coordination Gap — Large language models struggle with multi-agent coordination, and GRPO does not improve it; training methodology is the core bottleneck.
• Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek R1 enhances LLM reasoning via pure reinforcement learning and rule-based rewards, fostering autonomous problem-solving without supervised data.
• AI Red Teams and Adversarial Data Labeling with Redwood Research — Human red teaming is crucial for building highly robust AI models with extremely low false negative rates.
• From GRPO to SAMPO: Solving Training Collapse in Agentic RL — SAMPO stabilizes agentic RL training by optimizing four key policy dimensions to prevent catastrophic gradient issues.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.