Teaching models to forget: Selective unlearning with Amazon Nova

2026-07-06 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Cybersecurity & Data Privacy · Depth: Advanced, medium

Summary

Amazon Nova Customizable Content Moderation Settings (CCMS) addresses the challenge of foundation models over-deflecting legitimate business content due to stringent safeguards. This service allows approved customers to selectively adjust content moderation across four Responsible AI (RAI) pillars: Safety, Sensitive content, Fairness, and Security, while maintaining essential non-configurable controls. The underlying technology is Reverse Direct Preference Optimization (rDPO), a novel unlearning technique that utilizes Low-Rank Adaptation (LoRA) adapters. rDPO reverses the preference pair in the Direct Preference Optimization (DPO) objective, enabling models to unlearn specific policies while simultaneously generating high-quality alternative responses. Evaluation on Amazon Nova 2 Lite demonstrated substantial reductions in deflection rates, such as a 53.74 percentage point drop in Safety deflections (from 86.51% to 32.77%), alongside minimal degradation (less than 2 percentage points) in general model capabilities like instruction following, mathematical reasoning, and code generation. CCMS offers pre-trained rDPO LoRA adapters for deployment via Amazon Bedrock, with DPO training recipes available in Amazon SageMaker AI for custom research.

Key takeaway

For Machine Learning Engineers deploying foundation models with strict content moderation, Amazon Nova CCMS offers a critical solution. You can now selectively adjust model safeguards using rDPO-trained LoRA adapters, reducing over-deflection in legitimate business use cases without degrading core model capabilities. Consider utilizing pre-trained adapters via Amazon Bedrock for rapid deployment, or employ Amazon SageMaker AI for custom unlearning experiments to fine-tune specific policy areas. This enables precise control over model behavior for sensitive applications.

Key insights

Reverse Direct Preference Optimization (rDPO) selectively unlearns specific content moderation policies in foundation models, reducing over-deflection while preserving general capabilities.

Principles

• Unlearning can selectively modify model behavior.
• Reverse preference optimization guides unlearning to quality.
• LoRA adapters enable efficient, modular policy adjustments.

Method

rDPO reverses DPO's preference pair, training LoRA adapters to move models away from forgetting responses and towards high-quality target responses. This dual objective improves output quality and training efficiency for selective unlearning.

In practice

• Deploy pre-trained CCMS adapters via Amazon Bedrock.
• Explore custom unlearning with SageMaker AI DPO recipes.
• Combine CCMS with Bedrock Guardrails for layered safety.

Topics

• Foundation Models
• Content Moderation
• Selective Unlearning
• Reverse DPO
• LoRA Adapters
• Amazon Nova CCMS
• Responsible AI

Code references

• aws/sagemaker-hyperpod-recipes

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs — Post-training LLMs can regain output diversity without losing alignment by carefully constructing DPO preference data.
• Mechanism-Guided Selective Unlearning for RLVR-Induced Reasoning — Selective unlearning via mechanism-aligned targeting reduces collateral damage in RLVR-induced reasoning models.
• Quantization-Robust LLM Unlearning via Low-Rank Adaptation — Low-rank adaptation (LoRA) enables effective LLM unlearning that survives aggressive post-training quantization.
• Token-weighted Direct Preference Optimization with Attention — Using an LLM's intrinsic attention to weight tokens during preference optimization significantly improves alignment and performance.
• How to Fine-Tune an LLM: SFT, LoRA, QLoRA and DPO Explained — Fine-tuning adapts LLMs for specific tasks and behaviors using SFT, LoRA, QLoRA, and DPO.
• RepSelect: Robust LLM Unlearning via Representation Selectivity — RepSelect achieves robust LLM unlearning by selectively targeting forget-set-specific representations, preventing easy reversal.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.