Offline Contextual Bandits in the Presence of New Actions

2026-05-18 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Automated decision-making systems, like recommendation engines, typically use off-policy contextual bandits or off-policy learning (OPL) to select actions that maximize expected rewards from a predefined set. However, real-world applications often involve continuously evolving action spaces, where new actions emerge after data collection. This paper introduces a novel OPL method designed to address this challenge by leveraging action features. The approach first presents the Local Combination PseudoInverse (LCPI) estimator, which generalizes the PseudoInverse estimator for policy gradient estimation, controlling the trade-off between reward-modeling and data collection conditions for action features. Building on this, the authors propose Policy Optimization for Effective New Actions (PONA), an algorithm that combines LCPI, specialized for new action selection, with the Doubly Robust (DR) estimator, which is effective for existing actions. PONA is defined as a weighted sum of LCPI and DR estimators, allowing adjustment of new action selection proportions and demonstrating efficient new action selection while maintaining overall policy performance.

Key takeaway

For research scientists developing off-policy learning algorithms in dynamic environments, PONA offers a robust solution for handling continuously evolving action spaces. You should consider integrating PONA's LCPI and DR estimators to effectively select both existing and newly introduced actions, thereby improving the adaptability and performance of your automated decision-making systems, such as recommendation engines or search platforms.

Key insights

A new off-policy learning method, PONA, effectively selects both existing and novel actions in evolving environments.

Principles

• Action features enable learning for new actions.
• Combine specialized estimators for comprehensive action selection.

Method

PONA integrates the LCPI estimator, which generalizes PseudoInverse for new action selection, with the Doubly Robust (DR) estimator for existing actions, using a weighted sum to optimize both.

In practice

• Use LCPI for policy gradient estimation with action features.
• Implement PONA for dynamic recommendation systems.
• Adjust new action selection via PONA's weight parameter.

Topics

• Offline Contextual Bandits
• Off-Policy Learning
• New Actions
• Policy Optimization for Effective New Actions
• Local Combination PseudoInverse

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• The Fundamental Choice in Reinforcement Learning: On‑Policy vs. Off‑Policy — Reinforcement learning's core choice is between on-policy learning from current actions and off-policy learning from diverse experiences.
• Calibration-Gated LLM Pseudo-Observations for Online Contextual Bandits — LLM pseudo-observations can reduce cold-start regret in contextual bandits, but prompt design is critical.
• OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation — OPD-Evolver uses slow-fast co-evolution and on-policy self-distillation to create agents that holistically manage and learn from experience.
• Smaller Abstract State Spaces Enable Cross-Scale Generalization in Reinforcement Learning — Constraining RL agents to smaller abstract state spaces is crucial for achieving Out-of-Distribution generalization across complex tasks.
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents — TAPO corrects credit misassignment in tool-augmented agents by leveraging tool parameter determinism for improved policy optimization.
• Bandits for Efficient Experimentation: Adapting to Control Group, Preferences, and Context Drifts — The Dri-MED algorithm efficiently adapts multi-armed bandits to user preferences and context drifts, outperforming static baselines.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.