I Applied Andrej Karpathy’s Auto-research to Software Development

2026-04-19 · Source: AI Advances - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

Andrej Karpathy's auto-research pattern, where a Large Language Model (LLM) proposes changes and a harness verifies them in a loop, has been adapted for software development. This hybrid implementation, named scalar-loop, integrates the agent as a worker and defines invariants directly in Python code, rather than relying solely on prompts. This approach aims to overcome limitations of prompt-only systems, which can fail when agents encounter difficulties. In a practical application, the scalar-loop agent achieved a 95% reduction in bundle size, from 1492 characters to 70, without any sealed-file tampering, even when the agent attempted to quit after four tries.

Key takeaway

For AI Engineers developing autonomous agents for software tasks, consider implementing a hybrid approach like scalar-loop. Defining invariants directly in code, rather than relying solely on prompt engineering, can prevent agent "quitting" behaviors and lead to more robust, verifiable iterative improvements in metrics like bundle size or test coverage.

Key insights

Integrating LLM agents with code-defined invariants enhances autonomous software development iteration.

Principles

• LLM proposes, harness verifies
• Invariants in code prevent prompt-only failures

Method

The scalar-loop method uses an LLM agent to propose code changes, while Python-defined invariants in a verification harness determine whether to accept or revert the changes, iterating towards a desired metric.

In practice

• Reduce software bundle size
• Automate code refactoring
• Improve test coverage

Topics

• Andrej Karpathy's Auto-research
• Autonomous Software Development
• LLM-driven Iteration
• Hybrid AI Agents
• Code-based Invariants

Best for: AI Engineer, Software Engineer, Machine Learning Engineer

Related on AIssential

• Only the best are using them... — AI coding loops automate development by defining a verifiable end goal, allowing agents to self-iterate until completion.
• New review paper argues code is how AI agents think and act, not just what they produce — Code forms the executable, traceable, and stateful foundation for AI agent reasoning and action.
• Building a C compiler with a team of parallel Claudes — Parallel LLM agent teams can autonomously develop complex software projects, expanding the scope of AI-driven development.
• Components of a Coding Agent: How LLMs Become Software Engineers — Coding agents transform LLMs into dynamic, multi-step problem solvers through structured system design.
• LAI #129: Stop Babysitting Your Coding Agent — Optimizing AI workflows requires shifting from micro-management to autonomous agents and leveraging advanced techniques for cost and performance.
• SE Radio 711: Scott Hanselman on AI-Assisted Development Tools — Specificity and foundational knowledge are crucial for effectively steering AI-assisted development tools and managing inherent ambiguity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Advances - Medium.