Rank-Then-Act: Reward-Free Control from Frame-Order Progress

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

The Rank-Then-Act (RTA) framework enables learning control policies from expert video demonstrations without relying on environment rewards. RTA trains an offline Vision-Language Model (VLM) as a progress-based ordinal scorer, utilizing a Group Relative Policy Optimization (GRPO) objective on shuffled frame sequences to ensure temporal order recovery from visual semantics. Instead of direct scalar reward use, RTA employs a correlation-based reward function for reinforcement learning, computing the Spearman rank correlation between predicted progress rankings and true temporal indices within each interaction window. This approach provides a bounded, scale-invariant learning signal, decoupling reward learning from absolute calibration and facilitating stable transfer across tasks. Evaluated on discrete control benchmarks like PyBoy (Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld), RTA consistently matches or surpasses existing video-based reward learning methods and rank-based baselines, demonstrating robust cross-task reuse of a single pretrained scorer.

Key takeaway

For Machine Learning Engineers developing agents from expert demonstrations, Rank-Then-Act (RTA) provides a robust alternative to explicit reward engineering. By utilizing correlation-based rewards derived from video frame order, you can achieve stable policy learning and strong cross-task reuse of a single pretrained scorer. Consider implementing RTA's approach to decouple reward learning from absolute calibration, potentially simplifying complex task setups and improving transferability.

Key insights

Correlation-structured supervision from video-derived ordinal signals enables reward-free policy learning.

Principles

• Decouple reward learning from absolute calibration.
• Temporal ordering from visual semantics is key.
• Ordinal signals can suffice for policy learning.

Method

Train a VLM with GRPO on shuffled video frames to predict temporal order. Compute Spearman rank correlation between predicted and true order for RL reward.

In practice

• Use a single pretrained scorer across diverse tasks.
• Apply correlation-based rewards for stable transfer.
• Replace explicit reward design with video supervision.

Topics

• Rank-Then-Act
• Reward-Free Learning
• Vision-Language Models
• Policy Learning
• Video Demonstrations
• Ordinal Progress

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning — Optimal goal-conditioned actions depend on the goal's value gradient, not raw goal states.
• Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents — The Calibrate-Then-Act framework improves LLM agents' cost-aware exploration in uncertain environments.
• Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing — Decoupling visual perception, internal reasoning, and dialogue generation enables immersive, context-aware character role-playing.
• Reward Learning through Ranking Mean Squared Error — R4 uses a novel ranking MSE loss to learn RL rewards from ordinal human ratings, outperforming prior methods with less feedback.
• Policy-Conditioned Counterfactual Credit for Verifiable Reinforcement Learning of Long-Horizon Language Agents — Policy-conditioned counterfactual credit and validity gating significantly enhance verifiable long-horizon reinforcement learning for language agents.
• Reward-Aware Trajectory Shaping for Few-step Visual Generation — Preference alignment awareness in few-step generation can enable student models to surpass teacher performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.