Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Agon, a novel reinforcement learning method published on 2026-07-08, addresses limitations in current reasoning models that only grade final answers, often leading to verbose rather than improved thinking. This approach introduces two competing models that act as each other's graders. In alternating roles, one model drafts a solution while the other reads it and solves the problem, with rewards based on out-solving the rival. This competitive setup implicitly judges reasoning during training without requiring process labels or a separate reward model. By optimizing both models, Agon ensures each faces a progressively stronger opponent, a benefit single-model RL lacks. When deployed, the pair functions as a two-stage cascade. On the hard split of DeepMath with Qwen3, Agon doubles GRPO's pass@1 score, an eight-fold gain over an untrained Mixture-of-Agents pass, with results replicating across competitive-programming code and model families like Qwen3.5 and Gemma 4.

Key takeaway

For AI Scientists and Machine Learning Engineers developing advanced reasoning models, Agon offers a novel competitive RL approach that significantly enhances problem-solving by implicitly grading reasoning. You should explore integrating this two-stage cascade training to achieve substantial performance gains on complex tasks, as demonstrated by its doubling of pass@1 scores on DeepMath. Consider its application where current reward systems struggle to evaluate intermediate thought processes.

Key insights

Agon uses competitive reinforcement learning where two models implicitly grade each other's reasoning to improve problem-solving.

Principles

• Implicit reasoning judgment improves model thinking.
• Competitive training with progressively stronger rivals enhances performance.

Method

Agon involves two models attempting a problem; one drafts a solution, the other reads and solves, with rewards for out-solving the rival. This trains reasoning implicitly.

In practice

• Deploy as a two-stage cascade: one model drafts, the other answers.
• Apply to hard reasoning problems like DeepMath or competitive programming.

Topics

• Reinforcement Learning
• Large Language Models
• Competitive Training
• Reasoning Models
• DeepMath
• Qwen3

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning — RA-RFT teaches LMs analogical reasoning by training a retriever on reasoning benefit and fine-tuning with retrieved demonstrations.
• Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate — Tyler dynamically manages latent computation in LLMs, deciding when, what type, and how much to allocate for improved reasoning.
• Recursive Think-Answer Process to Stop "Thinking" Earlier — Integrating confidence-aware self-correction and RL-aware distillation improves LLM reasoning and efficiency.
• Learning to Reason with Insight for Informal Theorem Proving — Teaching LLMs to identify core techniques significantly improves their informal theorem proving capabilities.
• Select and Improve: Understanding the Mechanics of Post-Training for Reasoning — RL post-training enhances reasoning via strategy selection and improvement, driven by diverse pre-training and challenging RL data.
• Teaching Reasoning Models When to Stop: Data, Rewards, and Self-Aware Decoding — Efficient CoT reasoning requires balancing length reduction with accuracy, often by leveraging model confidence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.