From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Researchers from The Chinese University of Hong Kong and LIGHTSPEED introduce "Metric Freedom" ($F$), an a priori predictor for determining when distilling a Multi-Agent System (MAS) into a single-agent skill is beneficial. The study reveals that skill utility is governed by the evaluation metric's topological rigidity, not the task itself, with empirical outcomes showing skill lift ranging from a 28% improvement to a 2% degradation for the same task. They propose a two-stage adaptive distillation framework: Stage 1 selectively extracts tools and knowledge, discarding restrictive structures for "free" metrics ($F\approx 1$) while preserving exploration; Stage 2 targets computationally intensive iterative refinement exclusively toward "rigid" metrics ($F\lesssim 0.6$) to prevent trajectory-local overfitting. Evaluating across 4 tasks, 11 datasets, and 6 metrics, $F$ strongly predicts skill utility ($\rho{=}{-}0.62$, $p{<}0.05$). This adaptive agent matches or exceeds original MAS performance while reducing inference cost by 1.4-8\times and latency by up to 15\times.

Key takeaway

For research scientists evaluating or implementing multi-agent system (MAS) to single-agent skill distillation, you should first calculate the Metric Freedom ($F$) of your evaluation metric. This will predict whether skill augmentation will be beneficial or detrimental, guiding your distillation strategy to either preserve exploration for "free" metrics ($F\approx 1$) or apply structured guidance and iterative refinement for "rigid" metrics ($F\lesssim 0.6$), thereby optimizing performance and significantly reducing cost and latency.

Key insights

Skill distillation utility is determined by evaluation metric rigidity, not task complexity, quantified by Metric Freedom ($F$).

Principles

• Skill utility is a metric-level property.
• Structure helps rigid metrics, hurts free metrics.
• Tools extend capability without constraining reasoning.

Method

A two-stage adaptive distillation framework uses Metric Freedom ($F$) to selectively extract MAS components and guide iterative refinement, targeting rigid metrics ($F\lesssim 0.6$) for optimization.

In practice

• Compute Metric Freedom ($F$) from baseline runs.
• Discard MAS coordination structures for high-$F$ metrics.
• Refine skills iteratively only for low-$F$ metrics.

Topics

• Metric Freedom
• Skill Distillation
• Multi-Agent Systems
• Single-Agent LLMs
• Adaptive Distillation Framework

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Evidence Over Plans: Online Trajectory Verification for Skill Distillation — Robust agent skills must be posterior-based, distilled from empirical environment interaction rather than prior plans.
• The Illusion of Multi-Agent Advantage — Automated Multi-Agent Systems often incur high costs for superficial complexity, failing to outperform simpler Single-Agent Systems.
• Distill 5 AI Agents into ONE (w/ CODE) — Distilling multi-agent reasoning into a single LLM offers significant cost and efficiency benefits over direct multi-agent deployment.
• Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems — Skill-MAS decouples experience retention from LLM parametric updates using an evolvable Meta-Skill for multi-agent systems.
• Skill Distillation — Skill distillation transfers procedural knowledge from large models to smaller, local models via inspectable markdown files.
• SIGMA: Skill-Incidence Graphs for Compositional Multi-Agent Design — SIGMA enables multi-agent systems to generalize by dynamically composing agents from reusable skills, improving robustness and performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.