Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentation

2026-05-27 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Natural Language Processing · Depth: Expert, medium

Summary

Learning to Label (L2L) is a new reinforced self-evolving framework designed for semi-supervised referring expression segmentation (SS-RES). This framework tackles the challenges of limited annotation and unreliable pseudo-labels in SS-RES by treating pseudo-label construction as a learnable decision-making process. L2L integrates a multimodal large language model to derive semantic-spatial priors, which are then instantiated as initial soft segmentation proposals. These proposals, combined with textual cues, serve as learnable guidance for a hierarchical segmentation network. To ensure stable learning, L2L employs a reinforced pseudo-label selection mechanism that adaptively rewards high-utility pixel-level supervision, leveraging both multimodal priors and model predictions. This joint optimization of the segmentation model and pseudo-labels progressively enhances label reliability. Extensive experiments on datasets like RefCOCO, RefCOCO+, and RefCOCOg demonstrate L2L's effectiveness and generalization, showing improvements over existing methods.

Key takeaway

For Machine Learning Engineers developing semi-supervised vision-language models, consider integrating a reinforced self-evolving framework like L2L. This approach allows your system to learn reliable pseudo-label construction, directly addressing data scarcity challenges. You should explore using multimodal large language models to generate initial semantic priors and implement adaptive reward mechanisms for pseudo-label selection. This strategy can significantly enhance segmentation accuracy and generalization on datasets such as RefCOCO, even with sparse supervision.

Key insights

A reinforced self-evolving framework improves semi-supervised referring expression segmentation by learning to construct reliable pseudo-labels.

Principles

• Pseudo-label generation can be a learnable decision process.
• Multimodal priors enhance segmentation guidance.
• Reinforcement learning can adaptively select high-utility supervision.

Method

L2L extracts semantic-spatial priors via MLLM, generating soft segmentation proposals. These guide a hierarchical network, while reinforced selection adaptively rewards high-utility pseudo-labels for joint optimization.

In practice

• Apply MLLMs for initial semantic-spatial priors.
• Use reinforcement learning for adaptive pseudo-label selection.
• Jointly optimize segmentation models and pseudo-labels.

Topics

• Referring Expression Segmentation
• Semi-supervised Learning
• Pseudo-labeling
• Reinforcement Learning
• Multimodal LLMs
• Vision-Language Models

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• A Review of Pseudo-Labeling for Computer Vision — Pseudo-labeling extends beyond semi-supervised learning to enhance self-supervised and unsupervised methods.
• SSL-R1: Self-Supervised Visual Reinforcement Post-Training for Multimodal Large Language Models — SSL-R1 uses self-supervised visual puzzles to provide scalable, verifiable rewards for MLLM reinforcement learning.
• Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization — Large language models can learn and reproduce individual annotator explanation styles from human label variation.
• SegMix:Shuffle-based Feedback Learning for Semantic Segmentation of Pathology Images — Shuffle-based feedback learning improves weakly supervised semantic segmentation in pathology by adapting to multi-scale image characteristics.
• Accuracy Improvement of Semi-Supervised Segmentation Using Supervised ClassMix and Sup-Unsup Feature Discriminator — Improved semi-supervised segmentation uses supervised ClassMix and a feature discriminator to enhance accuracy and mitigate pseudo-label inaccuracies.
• Domain-Aware Hierarchical Contrastive Learning for Semi-Supervised Generalization Fault Diagnosis — DAHCL improves semi-supervised fault diagnosis by correcting pseudo-label bias and leveraging uncertain unlabeled data through domain-aware and hierarchical contrastive learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.