Autoresearch in the Wild: A Survey of Real-World Applications

2026-03-24 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, medium

Summary

Andrej Karpathy's autoresearch, released in January 2025, is an LLM agent-driven loop that autonomously modifies code, executes it, evaluates metric improvements, and then commits or reverts changes. This article surveys real-world applications of this pattern, drawing from the "awesome-autoresearch" repository. Initial applications included optimizing nanoGPT training code, where it found 20 improvements overnight. Shopify used it to optimize its Liquid template engine, achieving 53% faster parse+render and 61% fewer memory allocations. Other documented uses span GPU kernel optimization (18 TFLOPS to 187 TFLOPS), voice agent prompt engineering (score 0.728 to 0.969), and sports analytics (baseball pitch speed R-squared from 0.44 to 0.78). The pattern also extends to self-play domains with "autoevolve," which placed 6th out of 83 in the Game AI Cup by optimizing game bots through competitive evaluation.

Key takeaway

For AI scientists and engineers seeking to enhance system performance or discover novel solutions, consider implementing autoresearch for automated optimization. Your teams can apply this pattern to fine-tune LLM training, optimize critical infrastructure like template engines, or even explore complex scientific models. Be mindful of evaluation function design, as demonstrated by the tennis prediction case, to ensure true improvement and avoid reward hacking. Explore existing tooling like pi-autoresearch or autoevolve for specific use cases.

Key insights

Autoresearch leverages LLM agents to autonomously optimize code and configurations across diverse domains by iteratively modifying, running, and evaluating changes.

Principles

• Iterative modification and metric-based decision-making drives autonomous optimization.
• LLM agents can effectively navigate complex search spaces like CUDA kernels.
• Hybrid approaches combine LLM structural search with classical parameter optimization.

Method

The core autoresearch loop involves an LLM agent modifying code/config, executing it to measure a metric, deciding to commit or revert based on improvement, and repeating the process.

In practice

• Apply autoresearch to optimize existing codebases for performance metrics.
• Use self-play variants for competitive or adversarial AI tasks.
• Design robust evaluation functions to prevent reward hacking.

Topics

• Autoresearch
• LLM Agents
• Code Optimization
• Prompt Engineering
• Self-Play Algorithms

Code references

• WecoAI/awesome-autoresearch
• karpathy/autoresearch
• Shopify/liquid
• davebcn87/pi-autoresearch
• RightNow-AI/autokernel

Best for: NLP Engineer, AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, AI Researcher

Related on AIssential

• Let the AI Do the Experimenting — Autonomous AI agents can iteratively optimize complex problems by experimenting, measuring, and adapting code.
• Environment-Grounded Automated Prompt Optimization for LLM Game Agents — The framework automatically optimizes LLM agent prompts in interactive environments using an evolutionary loop and a multi-agent decomposition.
• Bilevel Autoresearch: Meta-Autoresearching Itself — An LLM-driven bilevel autoresearch framework can autonomously optimize its own search mechanisms for improved performance.
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery — AI agents struggle significantly with complex scientific literature discovery, achieving less than 10% accuracy on dedicated benchmarks.
• TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration — TREX automates LLM fine-tuning through a multi-agent, tree-based exploration system, outperforming human experts on specific tasks.
• Karpathy’s Autoresearch for Agent Engineering — AutoAgent autonomously optimizes AI agent harnesses through meta-agent experimentation and "model empathy."

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.