Agent Lightning: Adding reinforcement learning to AI agents without code rewrites

2025-12-11 · Source: Microsoft Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Advanced, medium

Summary

Microsoft Research Asia – Shanghai has introduced Agent Lightning, an open-source framework designed to integrate reinforcement learning (RL) into AI agents without requiring extensive code rewrites. LLM-based agents often struggle with complex, multi-step tasks, and while RL can enhance performance, its adoption is hindered by the need for significant code modifications. Agent Lightning addresses this by decoupling agent execution from model training, converting agent experiences into a standardized state-action sequence for RL. The framework employs a hierarchical RL approach, where its LightningRL algorithm assigns rewards to individual LLM requests within a multi-step task, making it compatible with existing single-step RL algorithms like PPO or GRPO. Evaluated across Text-to-SQL (LangChain), Retrieval-augmented generation (OpenAI Agents SDK), and Mathematical QA (AutoGen) scenarios, Agent Lightning consistently demonstrated performance improvements, such as enhanced SQL accuracy and better search query generation.

Key takeaway

For NLP Engineers and AI Scientists building or deploying LLM-based agents for complex, multi-step tasks, Agent Lightning offers a streamlined path to performance improvement. You should consider integrating this open-source framework to leverage reinforcement learning, as it allows for significant agent optimization without extensive code rewrites. This approach can enhance accuracy in areas like SQL generation, multi-hop Q&A, and tool use, making your agents more robust and efficient.

Key insights

Agent Lightning enables reinforcement learning for AI agents with minimal code changes by decoupling execution from training.

Principles

• Decouple agent execution from RL training.
• Standardize agent experience into state-action sequences.
• Use hierarchical RL for multi-step tasks.

Method

Agent Lightning converts agent execution into a sequence of LLM calls, each treated as an action with an assigned reward via a credit assignment module, enabling hierarchical RL training with single-step algorithms.

In practice

• Integrate RL into existing agents via Agent Lightning API.
• Optimize multi-step LLM tasks like RAG or Text-to-SQL.
• Scale agent training by decoupling components.

Topics

• AI Agents
• Reinforcement Learning
• Agent Lightning
• Hierarchical RL
• Multi-step LLM Tasks

Code references

• microsoft/agent-lightning

Best for: NLP Engineer, AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Agent RL Training Frameworks: 10 Open-source Tools to Know — Open-source agent RL training frameworks offer diverse solutions for building and optimizing AI agents across various task complexities and deployment scales.
• Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution — LLMs can effectively orchestrate RL skills for complex multi-agent coordination, achieving human-like adaptability.
• Can we build elite search agents without the massive industrial RL pipelines? — OpenSeeker-v2 aims to build elite search agents without industrial-scale RL pipelines.
• Microsoft’s open-source SkillOpt automatically upgrades AI agent skills without touching model weights — SkillOpt applies deep-learning optimization to text-based agent skills, enabling autonomous, verified performance improvements without altering model weights.
• AEL: Agent Evolving Learning for Open-Ended Environments — AEL enables LLM agents to learn from experience by dynamically selecting memory retrieval policies and integrating reflective causal insights.
• Meet KARL: A Faster Agent for Enterprise Knowledge, powered by custom RL — Custom reinforcement learning can drastically improve enterprise agent performance while reducing costs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Microsoft Research.