Active Inference as a Convex Markov Decision Process

2026-07-23 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

Active Inference (AIF) for adaptive behavior, which minimizes expected free energy (EFE), is framed as a convex Markov decision process (MDP) for closed-loop control policies. This formulation reveals that EFE minimization combines linear pragmatic terms, equivalent to reward maximization in a latent MDP, with a nonlinear epistemic value component, specifically the negative state-marginal entropy. This perspective allows the application of convex MDP theory and algorithms to AIF. The paper introduces MD-AIF, a mirror descent algorithm that locally linearizes the EFE objective, yielding a policy-dependent reward compatible with actor-critic methods and dynamic programming. MD-AIF demonstrates an O(1/K) convergence rate. Experiments on 5x5 and 10x10 gridworlds show MD-AIF achieves lower EFE and faster model learning through broader state coverage compared to standard reinforcement learning and EFE gradient descent. The work also addresses the performative nature of AIF when world-model learning is coupled with policy optimization, proving the existence of a performatively stable policy-model pair.

Key takeaway

For AI Scientists and Research Scientists developing Active Inference agents, understanding EFE minimization as a convex MDP is crucial. This framework allows you to apply robust optimization techniques like MD-AIF, which converges at an O(1/K) rate and promotes broad state coverage. You should consider implementing MD-AIF to improve the efficiency and exploration capabilities of your AIF models, especially in environments where model learning and policy adaptation are intertwined, leading to performative effects.

Key insights

Active Inference's EFE minimization is a convex MDP, enabling robust optimization via mirror descent and performative learning.

Principles

• EFE minimization combines linear pragmatic terms with a convex epistemic nonlinearity.
• Policy-dependent rewards can drive broad state coverage and accelerate model learning.
• Coupling model learning with policy optimization creates performative reinforcement learning.

Method

MD-AIF algorithm locally linearizes the EFE objective using mirror descent, transforming the convex MDP into a sequence of soft-MDP problems solvable by softmax dynamic programming.

In practice

• Implement MD-AIF for AIF agents to achieve O(1/K) convergence.
• Utilize policy-dependent rewards to enhance state exploration in gridworld environments.
• Consider performative RL frameworks when designing AIF agents with adaptive world models.

Topics

• Active Inference
• Convex Markov Decision Process
• Expected Free Energy
• Mirror Descent
• Reinforcement Learning
• Performative Learning

Best for: AI Scientist, Research Scientist

Related on AIssential

• Active Inference as a Convex Markov Decision Process — Active Inference's expected free energy minimization can be formulated as a convex Markov decision process.
• What Type of Inference is Active Inference? — Active inference planning requires combining planning and epistemic entropy corrections for a full variational characterization and effective message-passing.
• Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration — MDPO enhances differentiable planning by adaptively injecting stochastic noise for improved exploration in complex optimization landscapes.
• Provable Offline Reinforcement Learning for Structured Cyclic MDPs — CycleFQI offers a modular, theoretically robust offline RL solution for cyclic MDPs with heterogeneous stages.
• Hierarchical Active Inference using Successor Representations — Hierarchical active inference with successor representations enables efficient, scalable planning in complex, dynamic environments.
• Variational Proximal Policy Optimization — $ extsc{VP}_2 extsc{O}$ uses variational inference and MoE to improve PPO's stability and exploration in RLHF.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.