Let the AI Do the Experimenting

2026-04-28 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, long

Summary

The `autoresearch` framework, initially developed by Andrej Karpathy and extended by Shopify as `pi-autoresearch`, enables large language models (LLMs) to autonomously experiment, measure impact, and iterate on code to optimize a given objective. This article details an experiment using `pi-autoresearch` to solve a marketing budget optimization task with a $30 million budget, aiming to maximize revenue. The LLM-driven agent, starting from a greedy baseline yielding $107.9 million, explored various strategies including exact knapsack solvers and dynamic programming. It converged to an optimal solution of $110.16 million in revenue within five iterations. When additional constraints for customer support contacts (≤5K) and contact rate (≤0.042) were introduced, the agent reformulated the problem using an exact Mixed-Integer Linear Programming (MILP) solver, achieving $109.87 million revenue while satisfying all new conditions. The process highlights the agent's ability to systematically test hypotheses and converge on optimal solutions, even for complex, constrained problems.

Key takeaway

For Data Scientists and AI Engineers tasked with optimizing product KPIs under tight deadlines, `pi-autoresearch` offers a compelling approach to systematically explore and validate solutions. You can delegate iterative experimentation to an autonomous agent, freeing up time for more strategic work. Ensure your problem has clear, measurable objectives and well-defined constraints to maximize the agent's effectiveness, and always maintain human supervision to validate the practical feasibility of AI-generated "optimal" solutions.

Key insights

Autonomous AI agents can iteratively optimize complex problems by experimenting, measuring, and adapting code.

Principles

• Clear objectives enable autonomous optimization.
• Iterative experimentation drives continuous improvement.
• Human oversight validates AI-generated solutions.

Method

Define metrics and constraints, measure baseline, then an LLM agent proposes, tests, and discards/keeps code changes in a loop until convergence or limits are met.

In practice

• Use `pi-autoresearch` for KPI optimization.
• Set `maxIterations` to control token costs.
• Provide a baseline for agent initialization.

Topics

• autoresearch
• LLM Agents
• Marketing Budget Optimization
• Knapsack Problem
• Mixed-Integer Linear Programming

Code references

• karpathy/autoresearch
• karpathy/nanochat
• davebcn87/pi-autoresearch
• miptgirl/autoresearch_marketing_campaign

Best for: Data Scientist, AI Engineer, MLOps Engineer

Related on AIssential

• Autoresearch in the Wild: A Survey of Real-World Applications — Autoresearch leverages LLM agents to autonomously optimize code and configurations across diverse domains by iteratively modifying, running, and evaluating changes.
• AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks? — Long-horizon AI agent success hinges on persistent iteration and empirical feedback, not just initial solution quality.
• Karpathy’s Autoresearch for Agent Engineering — AutoAgent autonomously optimizes AI agent harnesses through meta-agent experimentation and "model empathy."
• Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents — Proactive, in-execution review by a specialized agent improves tool-calling agent reliability and performance.
• Researchers let AI Agents Optimize LLM Reasoning and Cut Tokens by 70% — AI agents can autonomously optimize LLM reasoning strategies, significantly reducing token usage while maintaining accuracy.
• I Applied Andrej Karpathy’s Auto-research to Software Development — Integrating LLM agents with code-defined invariants enhances autonomous software development iteration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.