RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences

2026-07-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

RENEW introduces a novel approach to mitigate model exploitation in world models, which are crucial for sample efficiency in offline reinforcement learning but susceptible to issues in low data coverage areas. Traditional solutions, such as collecting expensive expert demonstrations or employing conservative algorithms, often limit generalization or are impractical. RENEW proposes Dynamics Learning from Human Feedback (DLHF), a method that uses human preferences over imagined rollouts to identify and correct egregious dynamics hallucinations. DLHF formalizes this through a Bradley-Terry preference loss applied to trajectory log-likelihoods under a learned dynamics model. Recognizing the sample inefficiency of naive DLHF, RENEW incorporates epistemic uncertainty to strategically focus finetuning on the most exploitable model regions. Evaluations across Jumanji and classic control environments demonstrate that RENEW significantly enhances sample efficiency, prevents catastrophic forgetting, and effectively reduces exploitation in pretrained world models, thereby validating the use of human preferences for direct world model dynamics supervision.

Key takeaway

For Machine Learning Engineers developing offline reinforcement learning systems, if you are encountering model exploitation in world models, consider integrating human preferences. RENEW demonstrates that utilizing human feedback, particularly when focused by epistemic uncertainty, can efficiently repair dynamics hallucinations and reduce exploitation. This approach offers a practical alternative to expensive expert demonstrations or overly conservative algorithms, allowing you to build more robust and generalizable world models.

Key insights

RENEW repairs world model exploitation in offline RL by using human preferences over imagined rollouts, guided by epistemic uncertainty.

Principles

• World models exploit thin data coverage.
• Human preferences spot dynamics hallucinations.
• Epistemic uncertainty guides targeted finetuning.

Method

RENEW formalizes Dynamics Learning from Human Feedback (DLHF) using a Bradley-Terry preference loss over trajectory log-likelihoods. It then employs epistemic uncertainty to focus DLHF finetuning on the most exploitable model regions.

In practice

• Supervise world model dynamics with preferences.
• Use epistemic uncertainty for efficient finetuning.
• Reduce exploitation in model-based offline RL.

Topics

• World Models
• Offline Reinforcement Learning
• Model Exploitation
• Human Feedback
• Epistemic Uncertainty
• RENEW Algorithm

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Imperfect World Models are Exploitable — Imperfect world models in RL are inherently exploitable, leading to suboptimal policy choices.
• REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning — REGEN distills LLM knowledge using recycled replay memory and offline RL, decoupling training for cost efficiency.
• Recreating 180 Years of Chess Style with Policy-Only Fine-Tuning — Recreating historical play styles requires policy-only fine-tuning, a "social layer" for non-move behaviors, and rigorous validation against historical data.
• Current World Models Lack a Persistent State Core — Current world models lack persistent state evolution when unobserved, a critical gap requiring new design objectives beyond surface fidelity.
• Reinforcement Learning with Human Feedback (RLHF) in 4 minutes — RLHF uses human feedback and reinforcement learning to align LLM outputs with human preferences.
• BRICKS-WM: Building Reusability via Interface Composition Kinetics for Structured World Models — BRICKS-WM enables modular, reusable world models in MBRL by functionally separating agent and environment dynamics.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.