Joint Learning of Experiential Rules and Policies for Large Language Model Agents

2026-06-25 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Joint Learning of Experiential Rules and Policies for LLM Agents (JERP) addresses a key challenge for large language model agents operating in multi-step interactive environments: effectively utilizing accumulated interaction experience. Traditional approaches either store experience as external natural-language rules, which are interpretable but can become misaligned with an evolving policy, or update model parameters directly, offering broad policy improvement but limited local error correction in sparse-reward settings. JERP innovates by jointly updating a long-term experiential-rule pool and the agent's policy using the same interaction trajectories. During decision-making, JERP retrieves relevant rules to condition the agent alongside its interaction history. Post-episode, it optimizes the policy and refines the rule pool by comparing current rollouts against successful reference trajectories. This integrated approach ensures the rule pool remains synchronized with the evolving policy, gradually embedding stable and effective behaviors into the model itself. Experiments on AlfWorld and WebShop demonstrate consistent improvements in decision performance for complex interactive tasks.

Key takeaway

For AI Scientists and Machine Learning Engineers developing LLM agents for complex, multi-step interactive environments, you should consider JERP's joint learning mechanism. This approach effectively addresses the challenge of leveraging accumulated experience by aligning dynamic rule pools with policy updates. Implementing JERP can lead to more stable and effective agent behaviors, as demonstrated by consistent performance gains on benchmarks like AlfWorld and WebShop, enhancing both local error correction and overall policy improvement.

Key insights

JERP jointly learns experiential rules and policies from shared trajectories, aligning them for improved LLM agent performance.

Principles

• Coupling rule pools with policy updates ensures alignment.
• Experience can be absorbed into the model over time.
• Joint optimization improves local and broad policy aspects.

Method

JERP retrieves task-relevant rules to condition agents at decision time. Post-episode, it optimizes the policy and revises the rule pool by comparing current rollouts with successful trajectories.

In practice

• Apply JERP to multi-step interactive tasks.
• Use AlfWorld or WebShop for evaluation.
• Integrate rule-based conditioning with policy updates.

Topics

• LLM Agents
• Experiential Learning
• Policy Optimization
• Rule-based Systems
• Multi-step Environments
• AlfWorld
• WebShop

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• COMAP: Co-Evolving World Models and Agent Policies for LLM Agents — COMAP co-evolves world models and agent policies through closed-loop interaction for adaptive LLM agents.
• Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddings — MMSA combines model-based MARL with joint state-action representation learning for enhanced coordination and sample efficiency.
• Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning — TAO-RL stabilizes agentic reinforcement learning by filtering tool-use trajectories and guiding exploration with entropy for efficient LLM policy optimization.
• GEPA: How to Let an LLM Rewrite Its Own Prompts (and When It Actually Helps) — GEPA optimizes LLM prompts by having an LLM reflect on execution traces in natural language, outperforming scalar-reward RL with fewer rollouts.
• Learning and Reusing Policy Decompositions for Hierarchical Generalized Planning with LLM Agents — Dynamic policy learning with reusable components significantly boosts LLM agent performance and efficiency across diverse tasks.
• ExpSeek: Self-Triggered Experience Seeking for Web Agents — ExpSeek uses step-level entropy to proactively trigger and generate tailored experience guidance for web agents, significantly improving performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.