ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

ARCO (Adaptive Rubric CO-evolution) is a novel framework designed to improve multi-step LLM-based agents by providing interpretable, step-level process rewards. Unlike existing methods that rely on opaque scalar or trajectory-level rubric rewards from frozen, closed-source judges, ARCO employs a same-scale open-source model "μ" with a shared backbone. This model features a generation head for producing per-step natural-language criteria and a score head for predicting rubric-conditioned step-level rewards. A trajectory decomposition constraint links step rewards to the terminal outcome, enabling credit assignment without step-level labels. ARCO and the policy "π" are jointly updated on on-policy data, allowing the rubric content and scoring function to co-evolve. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue, using Qwen3-4B and Llama-3.2-3B backbones, demonstrate ARCO's superior Exact Match (EM) performance over strong baselines and confirm its rubrics are step-specific and robust.

Key takeaway

For AI Scientists and Machine Learning Engineers developing multi-step LLM agents, consider implementing co-evolutionary, step-level rubric-based reward models like ARCO. This approach provides granular, interpretable feedback, addressing the opacity and credit assignment issues of scalar rewards. By allowing evaluation criteria and scoring to adapt alongside the agent's policy, you can achieve superior performance and gain actionable insights into agent behavior, especially when using open-source backbones.

Key insights

Co-evolving step-level rubrics and scoring functions with the policy enhances multi-step LLM agent interpretability and performance.

Principles

• Reward models for multi-step agents should be rubric-based.
• Evaluation should assign per-action, step-level scores.
• Evaluation criteria and scoring functions must co-evolve with the policy.

Method

ARCO uses a shared-backbone model "μ" with a generation head for per-step criteria and a score head for rubric-conditioned step rewards. It jointly updates "μ" and policy "π" on on-policy data, using trajectory decomposition to learn step scores from terminal outcomes.

In practice

• Use K=3 criteria per step for optimal performance and efficiency.
• Leverage open-source, same-scale models for co-evolutionary reward modeling.
• Employ step-specific rubrics to diagnose agent behavior and failure modes.

Topics

• Multi-step LLM Agents
• Reinforcement Learning
• Rubric-based Rewards
• Co-evolutionary Training
• Multi-hop QA
• Process Reward Models

Code references

• zihangtian/ARCO

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation — EvoRubric enables LLMs to self-evolve rubrics for open-ended generation, overcoming static reward limitations.
• Agentic RL: Environments, Trajectories, and the Training — Training multi-step LLM agents requires scoring full interaction trajectories using an LLM-as-a-judge within a GRPO framework.
• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills — Skill Self-Play uses co-evolving agent skills to balance task diversity and reliable verification in LLM self-evolution.
• AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning — Persistent evaluation memory significantly improves rubric-based reward shaping for LLM fine-tuning in RL.
• Reward-Free Evolving Agents via Pairwise Validator — Pairwise LLM validators can replace costly scalar rewards for self-evolving agents, achieving competitive accuracy without labeling.
• Co-Evolving LLM Evaluators and Policies via DynamicRubric — Evaluators must co-evolve with policies to provide effective supervision for improving large language models.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.