Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

2024-01-30 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Omni-RRM is the first open-source rubric-grounded reward model designed to enhance multimodal large language models (MLLMs) by providing structured, multi-dimension preference judgments with justifications across text, image, video, and audio. It addresses limitations of existing vision-centric reward models (RMs) that often return opaque scalar scores and rely on costly human annotations. Omni-RRM is trained on Omni-Preference, a large-scale dataset of approximately 41K preference examples, built via a fully automated pipeline using strong teacher models for rubric-grounded rationales. The model achieves state-of-the-art accuracy on video (80.2% on ShareGPT-V) and audio (66.8% on Audio-HH-RLHF) benchmarks, and a 17.7% absolute gain over its base model on image tasks. It also improves downstream performance via Best-of-$N$ selection and transfers to text-only preference benchmarks.

Key takeaway

For AI Scientists and ML Engineers developing multimodal LLMs, Omni-RRM offers a robust solution for alignment. Its rubric-grounded, multi-dimensional feedback provides clear insights into model errors, surpassing opaque scalar scores. You should consider integrating Omni-RRM to enhance model interpretability, reduce reliance on costly human annotation, and achieve state-of-the-art performance across diverse modalities, including audio and video. This approach can significantly improve the reliability and trustworthiness of your MLLM deployments.

Key insights

Omni-RRM provides structured, rubric-grounded multimodal preference judgments without human annotation.

Principles

• Rubric-structured feedback improves MLLM alignment and interpretability.
• Automated data synthesis can replace costly human annotation for reward models.
• Unified reward models benefit lower-resource modalities through cross-modal supervision.

Method

Omni-RRM is trained in two stages: Supervised Fine-Tuning (SFT) for rubric-grounded output, then Group Relative Policy Optimization (GRPO) to sharpen discrimination on low-contrast pairs.

In practice

• Use Omni-RRM for fine-grained MLLM error resolution across modalities.
• Apply Best-of-$N$ selection with Omni-RRM to improve MLLM inference performance.
• Explore automated rubric-grounded data generation for custom alignment tasks.

Topics

• Omni-RRM
• Multimodal LLMs
• Reward Models
• Preference Alignment
• Rubric-Grounded Feedback
• Automated Data Synthesis
• Reinforcement Learning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• In-Context Reward Adaptation for Robust Preference Modeling — In-Context Reward Adaptation uses transformers and human response time to dynamically model diverse, unseen human preferences for robust RLHF.
• Reinforcement Learning with Robust Rubric Rewards — RLR³ extends RLVR with robust, criterion-level rubric verification for partially verifiable vision-language tasks, using LLMs and minimal exposure.
• Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users — Implicit user feedback (mouse/gaze) significantly enhances LLM alignment, outperforming explicit text-based methods.
• Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO — Continuous, variance-aware rubric rewards enable stable reinforcement learning for LLMs in complex, multi-criteria medical QA tasks.
• EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation — EvoRubric enables LLMs to self-evolve rubrics for open-ended generation, overcoming static reward limitations.
• RREDCoT: Segment-Level Reward Redistribution for Reasoning Models — RREDCoT improves CoT reasoning models by using the model itself for efficient segment-level reward redistribution, mitigating high variance from delayed rewards.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.