Not all uncertainty is alike: volatility, stochasticity, and exploration

2026-05-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

This paper introduces a novel framework demonstrating that distinct sources of environmental uncertainty—volatility (latent reward states drifting over time) and stochasticity (noisy outcome observations)—have opposing effects on optimal exploration in adaptive decision-making. While both increase posterior uncertainty, volatility enhances exploration by creating information gain, whereas stochasticity suppresses it by degrading observation utility. The authors formally establish this asymmetry by extending the Gittins index framework to Gaussian state-space bandits with latent dynamics. They further derive Cause-Aware Uncertainty-Sensitive Exploration (CAUSE), a closed-form index policy obtained via control-as-inference, which inherits these monotonicities. CAUSE empirically outperforms standard exploration strategies like Thompson sampling and UCB in environments with heterogeneous noise, and matches the optimal Gittins reference in rested bandit settings. The framework also predicts that miscalibrated noise inference can lead to reversed exploration patterns, offering insights into computational accounts of psychiatric conditions.

Key takeaway

For Machine Learning Engineers developing adaptive decision-making agents in dynamic environments, you should differentiate between volatility and stochasticity when designing exploration strategies. Relying solely on total uncertainty (as in UCB or Thompson sampling) can lead to suboptimal exploration, particularly over-exploring noisy arms. Implement CAUSE or similar cause-aware indices to ensure your agents prioritize information gain from environmental drift while minimizing unproductive exploration in highly stochastic settings, potentially improving regret performance in restless bandit problems.

Key insights

Optimal exploration is enhanced by volatility but suppressed by stochasticity, challenging the "more uncertainty, more exploration" paradigm.

Principles

• Volatility creates information gain.
• Stochasticity destroys information gain.
• Optimal exploration depends on uncertainty source.

Method

CAUSE is derived by casting action selection as posterior inference under an optimality constraint within the control-as-inference framework, yielding a closed-form index for restless bandits.

In practice

• Use CAUSE for restless multi-armed bandits.
• Distinguish volatility from observation noise.
• Investigate exploration reversals in psychiatric conditions.

Topics

• Multi-armed Bandits
• Exploration-Exploitation
• Volatility
• Stochasticity
• Control-as-Inference
• Gittins Index
• Computational Psychiatry

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Exploration and Exploitation: The Simple Yet Profound Logic at the Heart of Reinforcement Learning — Balancing exploration of new options with exploitation of known successes is central to learning under uncertainty.
• Best of both worlds: Stochastic & adversarial best-arm identification — Designing a single optimal learner for both stochastic and adversarial bandit best-arm identification is generally impossible.
• Robust Exploratory Stopping under Ambiguity in Reinforcement Learning — A robust RL framework for optimal stopping under ambiguity balances exploration and exploitation using $g$-expectation and BSDEs.
• Adaptive mine planning under geological uncertainty: A POMDP framework for sequential decision-making — Sequential belief updating transforms geological uncertainty into an active component of value creation in mine planning.
• Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration — MDPO enhances differentiable planning by adaptively injecting stochastic noise for improved exploration in complex optimization landscapes.
• Curiosity-Critic: Cumulative Prediction Error Improvement as a Tractable Intrinsic Reward for World Model Training — Curiosity-Critic improves world model training by rewarding cumulative prediction error reduction, separating learnable from unlearnable states.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.