How Autoresearch will change Small Language Models adoption

2026-03-10 · Source: philschmid.de - RSS feed · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, short

Summary

Autoresearch is a novel AI agent-driven optimization framework that autonomously edits machine learning training code, runs experiments, and iteratively improves model performance based on a specified metric. Developed by Karpathy, it operates with a fixed 5-minute time budget per experiment, a single-file code scope, Git for memory, and a binary keep/discard mechanism. Karpathy applied it to his nanochat GPT-2 training, achieving an 11% speed improvement (from 2.02 to 1.80 hours) over 700 experiments. Shopify CEO Tobi Lütke utilized a similar approach to train a 0.8B query expansion model overnight, which outperformed a previous 1.6B model by 19% after 37 experiments in 8 hours. The system is designed for small language models (SLMs) and tasks like search ranking, product categorization, and fraud scoring.

Key takeaway

For NLP Engineers or AI Scientists building domain-specific SLMs, autoresearch offers a powerful method to significantly accelerate model optimization. You should focus on developing robust, non-leaky evaluation metrics that reflect real-world performance, as this becomes the primary bottleneck when experiments run 100x faster. Consider integrating this autonomous optimization loop to achieve substantial performance gains with smaller models, potentially reducing computational costs and deployment complexity.

Key insights

Autoresearch enables autonomous, iterative optimization of ML models by an AI agent editing training code.

Principles

• Fixed time budget ensures comparable results.
• Git history guides agent's next steps.
• Binary keep/discard simplifies decision-making.

Method

An LLM agent edits a single training script, runs a short experiment, evaluates a metric, and commits changes if performance improves, repeating the cycle.

In practice

• Use for search ranking, product categorization.
• Start with open models like Gemma.
• Ensure robust, evolving evaluation pipelines.

Topics

• Autoresearch
• Small Language Models
• Model Optimization
• AI Agents
• Training Automation

Code references

• karpathy/autoresearch
• karpathy/nanochat
• tobi/qmd

Best for: NLP Engineer, AI Scientist, Research Scientist, Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• How to Build an Autonomous Machine Learning Research Loop in Google Colab Using Andrej Karpathy’s AutoResearch Framework for Hyperparameter Discovery and Experiment Tracking — Automate ML research loops in Colab for hyperparameter discovery and experiment tracking.
• Why Small Language Models Are Becoming the Default Choice for Private, Fast, Low-Cost AI in 2026 — SLMs are reshaping enterprise AI by prioritizing efficient, context-specific model deployment over sheer size.
• this EX-OPENAI RESEARCHER just released it... — AI agents can autonomously conduct machine learning research, improving model training and performance.
• Multi-Agent AutoResearch with Open Source Models — Specializing AI agents into distinct roles enhances complex task execution and allows for open-source model utilization.
• Andrej Karpathy's new open source 'autoresearch' lets you run hundreds of AI experiments a night — with revolutionary implications — AI agents can autonomously conduct and optimize experiments, accelerating research and discovery across diverse fields.
• Applying Karpathy's autoresearch to a 33M-token public transit dataset (14% improvement, replication notes) [P] — Autoresearch can improve language models on small, specialized datasets, but requires careful validation to avoid false positives.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by philschmid.de - RSS feed.