From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors

2025-08-07 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

A new study introduces the Benchmark-Assisted Guided evolutionary approach (BAG) for LLM-driven black-box optimization (BBO), demonstrating that providing high-quality algorithmic code examples significantly improves LLM performance. The research utilized AttnLRP to analyze token-wise prompt attribution, revealing that code-related content and associated strategy instructions exert the strongest influence on generated algorithmic code. Building on this, BAG integrates prior benchmark algorithms into an elitist search strategy, guiding LLMs towards promising search regions. Extensive experiments on 23 pseudo-Boolean optimization (pbo) problems and 24 continuous BBO (bbob) problems, using Gemini 2.0 Flash, GPT 5 Nano, and Qwen3 Coder Flash LLMs, show BAG consistently outperforms five state-of-the-art LLM-driven optimization methods, achieving up to a 14% improvement in average AUC on bbob problems. The findings highlight the value of fusing benchmark data to enhance the efficiency and robustness of LLM-driven BBO.

Key takeaway

For research scientists developing LLM-driven optimization methods, you should prioritize integrating established benchmark algorithms and high-quality code examples into your prompt designs. This strategy, as demonstrated by the BAG approach, significantly enhances the efficiency and robustness of algorithm generation, particularly for black-box optimization problems. Focusing on code-centric prompt guidance rather than solely linguistic instructions will lead to superior and more reliable algorithmic outcomes.

Key insights

Integrating benchmark code examples into prompts significantly enhances LLM-driven algorithm optimization performance and robustness.

Principles

• Code examples in prompts strongly influence LLM-generated algorithms.
• Benchmark algorithms can effectively guide LLM search spaces.
• Elitist search strategies are effective for LLM-driven optimization.

Method

The Benchmark-Assisted Guided (BAG) approach uses an (1+1) elitist search strategy, initializing with a promising benchmark algorithm and iteratively refining or creating new algorithms, with periodic benchmark algorithm injections.

In practice

• Use AttnLRP to analyze prompt component impact on code generation.
• Embed high-quality code examples in LLM prompts for algorithm design.
• Periodically inject benchmark algorithms to guide LLM evolutionary search.

Topics

• Large Language Models
• Automated Algorithm Design
• Black-box Optimization
• Prompt Engineering
• Evolutionary Algorithms

Code references

• BaronH07/BAG
• FeiLiu36/EoH
• Acquent0/LHNS
• XAI-liacs/LLaMEA
• zz1358m/MCTS-AHD-master

Best for: Research Scientist, AI Researcher, AI Scientist, Prompt Engineer

Related on AIssential

• Environment-Grounded Automated Prompt Optimization for LLM Game Agents — The framework automatically optimizes LLM agent prompts in interactive environments using an evolutionary loop and a multi-agent decomposition.
• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming — LLMs can complement traditional online judges by identifying subtle code errors, but effective error detection remains computationally intensive.
• What We’ve Learned From A Year of Building with LLMs — Successful LLM product development requires a holistic approach beyond demos, focusing on robust systems, data, and strategic iteration.
• Mutation Without Variation: Convergence Dynamics in LLM-Driven Program Evolution — LLM-driven program mutation inherently converges to limited structural forms, hindering open-ended exploration.
• The Model You Love Is Probably Just the One You Use — Developer preference for LLMs is often driven by familiarity and integration ease, not objective superiority.
• Autoresearch in the Wild: A Survey of Real-World Applications — Autoresearch leverages LLM agents to autonomously optimize code and configurations across diverse domains by iteratively modifying, running, and evaluating changes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.