ProPlay: Procedural World Models for Self-Evolving LLM Agents

2026-06-11 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

ProPlay is a novel procedural world model designed to enhance self-evolving LLM agents, particularly in partially observable environments where active exploration and learning from limited feedback are crucial. Unlike existing LLM-agent methods that often fail to continuously refine their internal understanding of environment dynamics, ProPlay introduces "procedure-level preplay." This mechanism allows agents to rehearse future procedural paths by leveraging learned world knowledge. It abstracts successful trajectories into "procedures" and organizes them within a "procedure graph" that maps causal transitions between task stages. Each transition in this graph is is associated with a reliability record embedding, which estimates its task-specific contribution from past outcomes. Before each episode, ProPlay simulates these future procedural trajectories to provide structured soft guidance, and after execution, it refines the graph based on environment feedback. Experiments on public benchmarks demonstrate that ProPlay consistently improves both environment understanding and self-evolution capabilities compared to strong baselines. The code is available on GitHub.

Key takeaway

For AI Engineers developing self-evolving LLM agents in complex, partially observable environments, ProPlay offers a robust approach to improve agent autonomy. You should consider integrating procedural world models that enable preplay and continuous graph refinement. This method enhances environment understanding and self-evolution, reducing reliance on external supervision and accelerating agent adaptation. Explore the released code to evaluate its applicability to your specific agent architectures.

Key insights

ProPlay uses a procedural world model and graph to enable LLM agents to self-evolve by rehearsing and refining future task paths.

Principles

• Abstract successful trajectories into procedures.
• Organize procedures in a causal graph.
• Estimate task contribution via reliability records.

Method

ProPlay simulates future procedural trajectories as soft guidance before an episode, then refines its procedure graph using environment feedback after execution, continuously improving world understanding.

In practice

• Enhance LLM agent exploration in complex environments.
• Improve agent learning from sparse feedback.
• Develop agents that refine internal world models.

Topics

• LLM Agents
• Self-Evolving Agents
• Procedural World Models
• Procedure Graph
• Environment Understanding
• Reinforcement Learning

Code references

• antman9914/proplay

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast — FORGE enables LLM agents to self-evolve memory via population broadcast, significantly improving decision-making without weight updates.
• COMAP: Co-Evolving World Models and Agent Policies for LLM Agents — COMAP co-evolves world models and agent policies through closed-loop interaction for adaptive LLM agents.
• Continual Harness: Online Adaptation for Self-Improving Foundation Agents [R] — Self-improving agents emerge from iterative harness refinement and model-harness co-learning.
• AI Agents of the Week: Papers You Should Know About — Hybrid LLM architectures with self-reflection and persistent environments enhance autonomous agent planning and learning.
• The Sequence Radar #824: Last Week in AI: Sovereign Lobsters, Self-Coding Agents, and Gigawatt Factories — AI is rapidly shifting from conversational tools to autonomous agents and world models, driving massive infrastructure investment.
• OpenSkill: Open-World Self-Evolution for LLM Agents — OpenSkill enables LLM agents to self-evolve skills and verification signals using open-world data, free from target-task supervision.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.