TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

TOUR (Trajectory-level memOrization and Unlearning in offline RL) is a new benchmark designed to evaluate trajectory-level data deletion in offline Reinforcement Learning. Offline RL agents, trained on fixed behavioral trajectories, require robust methods for removing specific data post-training. Evaluating this unlearning is challenging due to factors like membership scores, residual memorization, and potential policy collapse. TOUR addresses these complexities by integrating trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing. Experiments across D4RL locomotion and an exploratory AntMaze extension revealed that common deletion baselines exhibit environment-dependent privacy-utility behavior. The benchmark also demonstrated that retraining and fine-tuning often offer stronger retained-utility references than uniform GA+Refit, and that a single likelihood-based membership score frequently overstates deletion quality when subjected to diverse attack families.

Key takeaway

For Machine Learning Engineers evaluating unlearning in offline RL systems, relying solely on single likelihood-based membership scores is insufficient and can misrepresent deletion quality. You should adopt a multi-attack privacy auditing framework, like TOUR's approach, incorporating diverse attack families and matched non-member controls. This ensures a more stable and accurate assessment of unlearning effectiveness, balancing privacy guarantees with retained policy utility, and guiding robust model deployment decisions.

Key insights

Evaluating offline RL unlearning requires multi-faceted auditing beyond single likelihood scores, considering diverse attacks and utility.

Principles

• Offline RL unlearning evaluation needs comprehensive auditing.
• Single likelihood scores can overstate deletion quality.
• Unlearning privacy-utility is environment-dependent.

Method

TOUR combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing to rigorously evaluate unlearning in offline RL.

In practice

• Use retraining/fine-tuning as strong utility references.
• Employ multi-attack auditing for unlearning evaluation.
• Calibrate unlearning conclusions with non-member controls.

Topics

• Offline Reinforcement Learning
• Data Unlearning
• Machine Unlearning Benchmarks
• Privacy Auditing
• D4RL
• Trajectory-level Deletion

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget — A new auditing framework reliably verifies machine unlearning effectiveness, revealing varying success across different unlearning methods.
• POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents — POLAR-Bench diagnoses LLM agent privacy-utility trade-offs, showing frontier models significantly outperform smaller, on-device counterparts.
• PURGE: Projected Unlearning via Retain-Guided Erasure — PURGE unlearns data by projecting gradients to preserve retain-set performance and erasing hidden representations towards retain distribution.
• When Unlearning Is Free: Leveraging Low Influence Points to Reduce Computational Costs — Unlearning can be optimized by identifying and excluding data points with negligible influence on model outputs.
• Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certification — Current machine unlearning evaluation methods are flawed, requiring new, robust, and oracle-free certification approaches for reliability.
• Interference-Aware Multi-Task Unlearning — Multi-task unlearning requires mitigating task- and instance-level interference to preserve retained performance and generalization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.