Towards Healthy Evolution: Exploring the Role and Mechanisms of Human-Agent Interaction in Self-Evolving Systems

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Self-evolving agents, while capable of autonomous improvement through self-play and self-generated learning signals, can experience capability degradation and safety drift. The ANCHOR (Agent Norm Correction through Human-like Oversight and Review) framework, an LLM-based system, simulates human supervision to deliver feedback at various phases of self-evolution. Evaluated on open-source self-evolving agent systems like AZR and R-Zero, ANCHOR significantly mitigates safety degradation while preserving stable performance on core coding and mathematical reasoning objectives. Experiments using Qwen2.5 and Qwen3 models (3B, 4B, 7B, 8B, 14B) showed that supervision over the output verification (exec) phase is most effective, with its removal causing a 14.3% average performance drop. Increasing supervision frequency beyond f ∈ [0.3,0.4] yields diminishing returns, suggesting low-to-moderate intervention is optimal.

Key takeaway

For AI Scientists and Machine Learning Engineers developing self-evolving agent systems, integrating human-like supervision is crucial to prevent safety drift and maintain performance. You should prioritize feedback mechanisms during the agent's output verification phase, as this intervention proves most impactful. Implement a low-to-moderate supervision frequency, around 30-40%, to achieve significant gains without incurring excessive overhead, balancing effectiveness with resource constraints.

Key insights

Human-like oversight, particularly on output verification, effectively guides self-evolving agents to mitigate safety drift and maintain performance.

Principles

• Autonomous self-evolution risks safety drift and capability degradation.
• Feedback on verifier's execution results is the most impactful intervention.
• Supervision benefits exhibit diminishing returns at higher frequencies.

Method

ANCHOR uses an LLM-based supervisor to provide evaluative feedback at specific self-evolution phases (task, plan, thought, output, exec) via system prompt updates, with a Bernoulli gate for random review.

In practice

• Prioritize feedback on the verifier's decision (exec phase).
• Implement low-to-moderate supervision frequency (e.g., 30-40%).
• Use LLMs to simulate human oversight for scalable experiments.

Topics

• Self-Evolving Agents
• Human-Agent Interaction
• LLM Supervision
• AI Safety
• Reinforcement Learning
• Agent Frameworks

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Towards Healthy Evolution: Exploring the Role and Mechanisms of Human-Agent Interaction in Self-Evolving Systems — Limited human-like LLM supervision, especially during output verification, effectively mitigates safety degradation in self-evolving agents while maintaining performance.
• Okay, this unleashed my agent — Self-evolving agents leverage distinct memory and skill management architectures for continuous improvement.
• Continual Harness: Online Adaptation for Self-Improving Foundation Agents [R] — Self-improving agents emerge from iterative harness refinement and model-harness co-learning.
• New agent framework matches human-engineered AI systems — and adds zero inference cost to deploy — Group-Evolving Agents (GEA) enable AI systems to autonomously improve by fostering collective learning and shared innovation among agent groups.
• COMAP: Co-Evolving World Models and Agent Policies for LLM Agents — COMAP co-evolves world models and agent policies through closed-loop interaction for adaptive LLM agents.
• How to Fix Your Agent's Amnesia: Lessons from Building a Self-learning Agent — Learning agents, unlike stateless ones, continuously adapt and improve performance by retaining and applying knowledge from past interactions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.