SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation
Summary
SearchEyes introduces a novel approach to training multimodal search agents for multi-hop reasoning, addressing the structural disconnect in existing pipelines that independently construct training data, search environments, and reward signals. It unifies these components through a "simulated search world" built upon a typed knowledge graph. The system proposes Perception-Knowledge Chains (PKC) to sample constrained multi-hop paths over the visual-knowledge intersection of Wikidata5M, preserving hop-level entity metadata to define a self-contained search world and step-level reward anchors. Furthermore, Hop-Anchored Policy Optimization (HaPO) reuses these anchors for efficient step-level credit assignment without requiring a separate process reward model. Experiments across six multimodal knowledge-intensive benchmarks demonstrate that SearchEyes achieves leading performance among open-source multimodal search agents, with SearchEyes-27B improving over the strongest open-source baseline by 6.2 points on average.
Key takeaway
For AI Scientists and Machine Learning Engineers developing multimodal search agents, SearchEyes offers a significant advancement in multi-hop reasoning. You should investigate its "simulated search world" framework, which unifies training data, environments, and reward signals using a typed knowledge graph. Adopting its Perception-Knowledge Chains (PKC) and Hop-Anchored Policy Optimization (HaPO) can lead to top-tier performance and more efficient reinforcement learning, as demonstrated by SearchEyes-27B's 6.2-point average improvement.
Key insights
SearchEyes unifies multimodal search agent training via a simulated knowledge graph world and anchored policy optimization.
Principles
- Unify training data, environments, and reward signals.
- Use typed knowledge graphs for simulated search worlds.
- Anchor rewards at hop-level for efficient credit assignment.
Method
SearchEyes constructs a simulated search world using a typed knowledge graph. It employs Perception-Knowledge Chains (PKC) for path sampling and Hop-Anchored Policy Optimization (HaPO) for step-level credit assignment.
In practice
- Develop multi-hop reasoning agents for complex queries.
- Integrate visual and knowledge data for enhanced search.
- Improve RL training efficiency with hop-level rewards.
Topics
- Multimodal AI
- Deep Search
- Knowledge Graphs
- Reinforcement Learning
- Multi-hop Reasoning
- SearchEyes
Best for: Research Scientist, AI Scientist, Machine Learning Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.