Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

This study investigates predicting the closed-loop performance of learned latent world models, crucial for Model Predictive Control (MPC) and Model-Based Reinforcement Learning (MBRL). Traditional validation metrics like loss and multi-step prediction RMSE often improve even as closed-loop performance degrades. The authors introduce a suite of structural validation-time diagnostics derived from optimal-control theory, applying them to Gymnasium's LunarLander v3 with shaped rewards. They trained an RSSM world model and used per-checkpoint CEM-MPC return as an oracle. Evaluating 40 metrics, the Reward Observability Fraction (ROF), measuring reward predictor dependence on the observable subspace, emerged as the strongest single predictor. Combining ROF with three structural regularizers, they developed the Composite Reward Observability Fraction (CROF) for offline checkpoint selection. A CROF-selected world model trained a model-based A2C policy that surpassed a model-free A2C baseline by ~24.5 return points, utilizing ~65x fewer real-environment interactions, and also powered a robust zero-shot CEM-MPC policy.

Key takeaway

For Machine Learning Engineers developing model-based reinforcement learning or MPC systems, selecting optimal world model checkpoints is critical. If you rely solely on validation loss or RMSE, you risk deploying models with collapsed closed-loop performance. You should implement the Composite Reward Observability Fraction (CROF) as an offline checkpoint selection score. This approach can significantly improve policy training, enabling model-based agents to achieve superior returns with substantially fewer real-environment interactions compared to model-free baselines.

Key insights

Validation metrics often fail to predict latent world model closed-loop performance; structural diagnostics like CROF offer a reliable offline selection method.

Principles

• Standard validation metrics can mislead for world model performance.
• Optimal-control theory offers relevant structural diagnostics.
• Reward Observability Fraction (ROF) is a strong performance predictor.

Method

Train an RSSM world model, evaluate 40 metrics against CEM-MPC return, identify ROF, then combine ROF with three structural regularizers into a Composite Reward Observability Fraction (CROF) for offline checkpoint selection.

In practice

• Use CROF for selecting world model checkpoints.
• Apply CROF to improve model-based A2C policy training.
• Drive zero-shot CEM-MPC policies with CROF-selected models.

Topics

• Latent World Models
• Model Predictive Control
• Model-Based Reinforcement Learning
• Offline Checkpoint Selection
• Reward Observability Fraction
• RSSM

Code references

• nsmoly/LunarLander_RSSM

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Curiosity-Critic: Cumulative Prediction Error Improvement as a Tractable Intrinsic Reward for World Model Training — Curiosity-Critic improves world model training by rewarding cumulative prediction error reduction, separating learnable from unlearnable states.
• A Predictive Law for On-Policy Self-Distillation From World Feedback — On-Policy Self-Distillation (OPSD) performance is predictably linear with the initial student-self-teacher performance gap, enabling pre-training outcome anticipation.
• ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control — Adversarial robustness for world models requires multi-level, temporal attack assessment beyond just action-space disruption.
• When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming — RLHF failures are dynamic, classifiable training events, not just final model pathologies.
• The RL Fine-Tuning Playbook: CoreWeave's Kyle Corbitt on GRPO, Rubrics, Environments, Reward Hacking — Reinforcement learning fine-tuning offers superior performance and efficiency over SFT by leveraging a model's inherent strengths.
• Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning — Preserving diverse policy neighborhoods in a stable latent space enhances continual reinforcement learning plasticity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.