Improving Composer through real-time RL

2026-03-26 · Source: Cursor Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, medium

Summary

Cursor has implemented a "real-time RL" approach to continuously improve its coding model, Composer, by using live inference tokens from user interactions as training signals. This method addresses the inherent train-test mismatch in simulated coding environments, particularly the difficulty in accurately modeling user behavior. By collecting billions of tokens from user interactions, distilling them into reward signals, and rapidly deploying updated checkpoints, Cursor can ship an improved version of Composer as frequently as every five hours. This rapid iteration allows the training data to remain on-policy and has led to measurable improvements, including a +2.28% increase in agent edits persisting in the codebase, a -3.13% reduction in dissatisfied follow-ups, and a -10.3% decrease in latency. The system also actively mitigates reward hacking by treating such instances as "bug reports" to refine the training system.

Key takeaway

For AI Scientists and Research Scientists developing coding assistants, adopting a real-time RL framework can significantly accelerate model improvement and reduce simulation-induced train-test mismatch. Your team should prioritize robust client-side instrumentation and rapid deployment pipelines to leverage live user data effectively, enabling continuous, on-policy model refinement and faster adaptation to real-world user needs.

Key insights

Real-time Reinforcement Learning (RL) using live user inference tokens effectively improves coding models and mitigates train-test mismatch.

Principles

• Real user feedback eliminates simulation modeling error.
• Rapid iteration keeps training data on-policy.
• Reward hacking reveals system vulnerabilities.

Method

Collect user interaction tokens, distill into reward signals, calculate model weight adjustments, run against eval suites (e.g., CursorBench) for regressions, and deploy updated checkpoints within hours.

In practice

• Instrument client-side for user interaction signals.
• Implement fast deployment paths for model updates.
• Monitor for reward hacking to refine reward functions.

Topics

• Real-time RL
• Coding Models
• Composer
• Train-Test Mismatch
• Reward Hacking

Best for: AI Scientist, Research Scientist, Machine Learning Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• Lessons from Trillion Token Deployments at Fortune 500s — Alessandro Cappelli, Adaptive ML — Reinforcement Learning systematically integrates feedback to accelerate LLM lifecycle, enabling production-ready, cost-effective, and performant models.
• Reward Hacking in Reinforcement Learning — Reward hacking exploits reward function flaws, leading to unintended behaviors in RL agents, especially LLMs.
• A technical report on Composer 2 — Continued pretraining and large-scale RL on realistic data significantly enhance coding model performance and efficiency.
• RL for Agents Workshop - Deep Dive on Training Agents with RL and Open Source — AI agents are evolving rapidly, enabling long-horizon tasks through advanced RL training, custom models, and robust evaluation.
• Reinforcement Learning with Human Feedback (RLHF) in 4 minutes — RLHF uses human feedback and reinforcement learning to align LLM outputs with human preferences.
• Real-Time Execution with Autoregressive Policies — Autoregressive policies can achieve real-time execution through tokenization horizon adjustment and constrained decoding, outperforming flow-matching.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Cursor Blog.