A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

2026-07-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Group Relative Policy Optimization (GRPO) applied to small language and vision-language model web agents (4B to 8B scale) does not credibly improve success rates on tasks the agent has largely mastered. Across 18 controlled runs varying learning rate, KL weight, seed, initialization, and clipping, moderate to high learning rates actually made performance credibly worse on the text track. This null finding holds robustly under paired testing, 25 evaluation seeds, 6 training seeds, recipe changes, and scaling to 8B. However, GRPO did increase success rates by 22 points on tasks with reachable rewards, indicating it helps only when there is significant performance headroom. The failure mechanism involves middle learning rates degrading the agent and high ones causing collapse, with distinct localization patterns; effective rank in late layers tracks capability at 4B but not 8B, highlighting scale-dependent effects.

Key takeaway

For machine learning engineers applying Group Relative Policy Optimization (GRPO) to small language or vision-language models, understand that it may not enhance performance on already-mastered tasks and can degrade results, especially with moderate to high learning rates. Prioritize GRPO for tasks where your base model has significant performance headroom, and meticulously test learning rate sensitivity to avoid detrimental effects.

Key insights

GRPO fails to improve small web agents on mastered tasks, only helping when significant performance headroom exists.

Principles

• GRPO's effectiveness is task-dependent, requiring performance headroom.
• Learning rate significantly impacts GRPO's outcome, from degradation to collapse.
• Effective rank in late layers can track capability, but this coupling is scale-dependent.

In practice

• Avoid GRPO on small L/VLMs for already-mastered tasks.
• Carefully tune learning rates when applying GRPO to web agents.
• Consider model scale when interpreting effective rank changes.

Topics

• Group Relative Policy Optimization
• Reinforcement Learning
• Language Models
• Vision-Language Models
• Web Agents
• Learning Rate Tuning
• Policy Optimization

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The ONLY DeepSeek GRPO/PPO video you'll EVER need (with examples and exercises) | RL Foundations — GRPO optimizes language models by adjusting rewards relative to group performance and constraining policy updates for stability.
• Training Agents 3: Reinforcement Learning — GRPO enables outcome-based agent training using relative scores from multiple completions, eliminating the need for a separate critic or reward model.
• Information-Consistent Language Model Recommendations through Group Relative Policy Optimization — GRPO can enforce information consistency in LLMs by optimizing for stable outputs across semantically equivalent prompts.
• GRPO without the maths: how an RL trainer nudges the weights — GRPO updates agent weights by comparing individual attempt scores to a group baseline, rewarding positive "advantage."
• A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism — GRPO only improves web agents when the sampled policy already outperforms the greedy one.
• GRPO from Scratch: Group Relative Policy Optimization in Python — GRPO simplifies RL training by eliminating the value model, using group responses as a baseline.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.