Co-Evolving LLM Evaluators and Policies via DynamicRubric

2026-07-22 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

DynamicRubric is a novel co-evolution framework designed to improve large language models (LLMs) by dynamically evolving evaluators alongside the policies they supervise. It addresses the bottleneck in post-training where improving policies lead to similar-quality responses, causing collapsed relative evaluator score gaps that yield weak policy supervision. The framework generates weighted binary rubric items for each candidate response set and aggregates judgments into response-level scores. Experiments with 8B LLM backbones demonstrate that DynamicRubric enhances evaluator performance and provides stronger policy supervision compared to baselines using a 70B reward model or a 235B static rubric generator. Policies optimized with DynamicRubric also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, handling tens of millions of requests daily and improving key online metrics.

Key takeaway

For Machine Learning Engineers optimizing LLMs with evaluator feedback, you should consider implementing dynamic evaluator-policy co-evolution frameworks like DynamicRubric. This approach directly addresses the issue of diminishing supervision signals as model quality improves, ensuring your policy updates remain effective. By adopting dynamic rubrics, you can achieve stronger policy supervision, leading to improved performance on tasks like verifiable reasoning and coding, and better online metrics in production environments.

Key insights

Evaluators must co-evolve with policies to provide effective supervision for improving large language models.

Principles

• Relative evaluator score gaps guide policy updates.
• Dynamic evaluators improve policy optimization signals.

Method

DynamicRubric generates weighted binary rubric items per candidate set, aggregating judgments into response-level scores for co-evolution.

In practice

• Deploy DynamicRubric for LLM post-training.
• Use dynamic rubrics to enhance policy supervision.
• Apply to reasoning and coding task improvements.

Topics

• LLM Evaluation
• Policy Optimization
• DynamicRubric
• Reward Models
• WeChat Search
• Post-training

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation — EvoRubric enables LLMs to self-evolve rubrics for open-ended generation, overcoming static reward limitations.
• Self-Evolving AI: LLM and Harness Together (RSI) — A small LLM can outperform larger models by co-evolving its policy and environment, including its own evaluators.
• Learning and Reusing Policy Decompositions for Hierarchical Generalized Planning with LLM Agents — Dynamic policy learning with reusable components significantly boosts LLM agent performance and efficiency across diverse tasks.
• ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents — Co-evolving step-level rubrics and scoring functions with the policy enhances multi-step LLM agent interpretability and performance.
• LLM-Driven Co-Evolutionary Automated Heuristic Design for Bi-Component Coupled Combinatorial Optimization — CoEvo-AHD uses LLMs and co-evolution to design heuristics for coupled combinatorial optimization problems, improving solution quality.
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments — Automated environment generation and verifiable rewards significantly boost LLM tool-use performance via reinforcement learning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.