Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents?

2026-07-08 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A study investigated optimal model capacity distribution in multi-agent large language model (LLM) search architectures, which typically decompose complex queries into sub-queries for parallel sub-agents. Researchers factorized hierarchical search into three roles: delegation (task decomposition), execution (retrieval and evidence extraction), and answer generation. Conducting controlled capacity sweeps on delegation and execution across five multi-hop QA benchmarks, the study found that role factorization consistently improved exact match (EM) scores from 4.5 to 8.6 points compared to single-agent baselines. Capacity sensitivity was asymmetric, with delegation backbone scaling yielding an ~11-point EM improvement, while execution sub-agent scaling only provided ~2.6 points, identifying decomposition as the primary bottleneck. Furthermore, a 1.7B-parameter executor, trained with quality-filtered trajectory distillation, achieved accuracy comparable to a frontier sub-agent while consuming 37% fewer sub-agent tokens.

Key takeaway

For Machine Learning Engineers designing multi-agent LLM systems, prioritize allocating computational resources to the delegation component. Your focus should be on enhancing the model capacity for task decomposition, as this is the primary performance bottleneck. You can then downsize your execution sub-agents, potentially using techniques like quality-filtered trajectory distillation, to achieve comparable accuracy with significantly fewer tokens, optimizing cost and efficiency.

Key insights

Capacity in hierarchical LLM search agents should prioritize delegation for decomposition over execution for retrieval.

Principles

• Hierarchical search agents improve exact match.
• Delegation capacity is crucial for performance.
• Smaller executors can match frontier sub-agents.

Method

The study factorized hierarchical search into delegation, execution, and answer generation roles. It conducted controlled capacity sweeps on delegation and execution axes using multi-hop QA benchmarks.

In practice

• Concentrate capacity on delegation roles.
• Downsize execution sub-agents.
• Apply trajectory distillation for executors.

Topics

• Large Language Models
• Multi-agent Systems
• Hierarchical Search
• Model Capacity Allocation
• Task Decomposition
• Trajectory Distillation

Code references

• QinnanCai0115/role-factorized-search

Best for: AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents? — Concentrating model capacity in the delegation role significantly improves hierarchical search agent performance, while execution can be downsized.
• Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution — LLMs can effectively orchestrate RL skills for complex multi-agent coordination, achieving human-like adaptability.
• Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose — Compositional skill routing for LLM agents requires decomposition, retrieval, and plan composition.
• AIエージェントが最適化プログラミングコンテストで初優勝 — AI agents can now win complex, real-time optimization contests by scaling LLM inference and self-learning.
• Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents — Experience compression unifies LLM agent memory, skills, and rules, crucial for scalable, long-horizon deployments.
• Scaling up multi-agent systems: an interview with Minghong Geng — Hierarchical and hybrid designs are key to scaling multi-agent systems for complex, long-horizon problems.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.