A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Group Relative Policy Optimization (GRPO) does not credibly enhance the skill of small language and vision-language model web agents (4B to 8B Qwen3-VL) on tasks they have largely mastered, according to a study on the MiniWoB benchmark. Across 18 controlled runs varying learning rate, KL weight, seed, initialization, and clipping, no configuration improved the supervised baseline's 49.1% success rate. Instead, moderate to high learning rates degraded performance, with a middle rate reducing success by 15 points and a high rate collapsing it to zero on the text track. A positive control experiment demonstrated GRPO's effectiveness, increasing success by 22 points (from 20.0% to 42.0%) on tasks where the sampled policy already outperformed the greedy one. The failure mechanism involves learning-rate-gated lesions: a middle rate destroys late-layer effective rank in attention and MLP blocks, while a high rate causes a distributed corruption of the readout. This rank-capability coupling is scale-dependent, holding at 4B but breaking at 8B.

Key takeaway

For Machine Learning Engineers deploying small language and vision-language web agents, if your agent has already mastered its tasks, you should first calculate the "Δ_head" (sampled vs. greedy success). If this gap is zero or negative, invest in improved supervision or distillation rather than applying GRPO, as it is unlikely to add new skills and high learning rates can credibly degrade your agent's performance.

Key insights

GRPO only improves web agents when the sampled policy already outperforms the greedy one.

Principles

• Reinforcement learning may only sharpen existing model capabilities, not add new ones.
• Effective rank in late layers can track model capability at smaller scales (e.g., 4B).
• High learning rates can cause distributed corruption, not localized damage, in models.

Method

The study used causal grafting by restoring component groups (attention, MLP, embedding) from initialization to identify which weights carry failure, separating correlation from causation.

In practice

• Measure "Δ_head = SR_sample - SR_greedy" before applying GRPO.
• Prioritize better supervision or distillation for competent small web agents.
• Avoid high learning rates in GRPO for small language models, especially on text tasks.

Topics

• Group Relative Policy Optimization
• Web Agents
• Language Models
• Vision-Language Models
• Reinforcement Learning
• Learning Rate Optimization
• Model Interpretability

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism — GRPO fails to improve small web agents on mastered tasks, only helping when significant performance headroom exists.
• Exploration and Exploitation Errors Are Measurable for Language Model Agents — A policy-agnostic framework quantifies LM agent exploration and exploitation errors in complex, partially observable environments.
• Improving Cross-Lingual Factual Recall via Consistency-Driven Reinforcement Learning — GRPO significantly improves cross-lingual factual recall and consistency in LLMs by fostering shared representations, outperforming SFT.
• Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models — Asymmetric token-level discounting in GRPO significantly stabilizes and improves reinforcement learning for language models.
• The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works — GRPO's z-scoring amplifies dense reward variance, causing LLM agent collapse; auxiliary losses are more effective.
• From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python — Reinforcement learning effectively utilizes hierarchical vulnerability penalties as training signals, outperforming supervised methods under distribution shift.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.