Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance

2026-05-14 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

FEST, a Few-Shot demonstration-guided Reinforcement Learning with Verifiable Rewards (RLVR) algorithm, significantly improves sample efficiency for Large Language Models (LLMs) on complex tasks like math and coding. Traditional RLVR struggles with difficult problems due to the challenge of generating correct rollouts, and prior demonstration-guided methods often require extensive Supervised FineTuning (SFT) data. FEST achieves strong performance using only 128 randomly selected demonstrations from an SFT dataset, outperforming baselines that use substantially more SFT data and even matching their performance with full datasets. Key to FEST's success are three components: a supervised signal, an on-policy signal, and decaying weights applied to the few-shot SFT dataset to mitigate overfitting during multi-epoch training.

Key takeaway

For AI Engineers developing LLMs for complex tasks, FEST offers a compelling approach to improve RLVR sample efficiency without the high cost of extensive SFT data. You should consider integrating FEST's three vital components—supervised signal, on-policy signal, and decaying SFT weights—to achieve strong performance with significantly fewer demonstrations, potentially reducing data acquisition expenses and training time.

Key insights

FEST enhances RLVR sample efficiency for LLMs using few-shot demonstrations, outperforming baselines with significantly less data.

Principles

• Combine supervised and on-policy signals for RLVR.
• Decay SFT weights to prevent few-shot overfitting.

Method

FEST integrates supervised and on-policy signals with decaying weights on a small, randomly selected few-shot SFT dataset to guide RLVR, improving sample efficiency for LLMs.

In practice

• Use 128 demonstrations for few-shot SFT.
• Implement decaying weights for SFT data.

Topics

• Reinforcement Learning with Verifiable Rewards
• Large Language Models
• Few-Shot Learning
• Supervised FineTuning
• Sample Efficiency

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• sGPO: Trading Inference FLOPs for Training Efficiency in RLVR — sGPO uses cheap inference to profile query difficulty, optimizing RLVR training by reducing wasted FLOPs.
• Accelerate LLM post training with W&B Serverless SFT — W&B Training simplifies LLM fine-tuning and RL iteration by providing serverless SFT and integrated artifact management.
• Reinforcement Learning without Ground-Truth Solutions can Improve LLMs — RiVER trains LLMs on score-based tasks without ground-truth, using calibrated reward shaping for general coding ability.
• LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking — LLMs trained with RLVR can exploit imperfect verifiers through reward hacking, bypassing genuine rule induction.
• RL Scaling Laws for LLMs — Scaling laws, while precise for LLM pretraining, are more complex and context-dependent in reinforcement learning.
• RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning — POPO improves LLM reasoning by efficiently leveraging effective off-policy data in RLVR, reducing computational overhead.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.