UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

UniICL introduces a systematized approach to unified multimodal in-context learning (ICL), addressing the sensitivity and non-monotonic efficacy observed in current models due to cross-modal interference. Researchers from Zhejiang University and collaborators developed a six-level capability-oriented taxonomy, categorizing demonstration roles from basic perception to high-order discernment. Guided by this framework, they constructed UniICL-760K, a large-scale corpus with 766,868 curated 8-shot ICL episodes across 15 subtasks, and UniICL-Bench for controlled evaluation. Additionally, the Context-Adaptive Prototype Modulator (CAPM), a lightweight, plug-and-play module, was proposed to stabilize few-shot adaptation. Evaluations on UniICL-Bench demonstrate that UniICL achieves highly competitive unified results, outperforming larger-parameter multimodal large language model baselines on most understanding ICL tasks and exhibiting enhanced stability.

Key takeaway

For AI Architects and Machine Learning Engineers developing unified multimodal models, this research underscores that effective in-context learning requires more than just adding demonstrations. You should adopt a capability-oriented approach to diagnose task-specific cognitive demands and consider architectural enhancements like the Context-Adaptive Prototype Modulator. This will stabilize few-shot adaptation, mitigate non-monotonic performance, and ensure consistent gains, especially for complex generative tasks where demonstration quality is critical.

Key insights

A capability-oriented taxonomy and adaptive modulation stabilize unified multimodal in-context learning across diverse cognitive demands.

Principles

• Multimodal ICL efficacy is non-monotonic and task-dependent.
• A six-level taxonomy structures ICL tasks by cognitive role.
• Generative ICL is more sensitive to demonstration quality.

Method

UniICL-760K uses a cascaded annotation and generative synthesis pipeline with DPP and intent-driven retrieval. CAPM employs decoupled encoding, low-rank transformation, adaptive routing, and element-wise gating.

In practice

• Diagnose ICL failures using a cognitive capability taxonomy.
• Apply CAPM for stable few-shot multimodal adaptation.
• Focus on high-quality demonstrations for generative ICL.

Topics

• Multimodal In-context Learning
• Capability Taxonomy
• Context-Adaptive Prototype Modulator
• Unified Multimodal Models
• Few-shot Learning
• ICL Benchmarking

Code references

• xuyicheng-zju/UniICL

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models — The survey unifies vision-language perception in MLLMs, proposing a five-stage evolution taxonomy and future directions for AGI.
• Free Lunch for Unified Multimodal Models: Enhancing Generation via Reflective Rectification with Inherent Understanding — UniRect-CoT improves UMM generation by activating inherent understanding for self-rectification during denoising.
• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification — A single discrete visual tokenizer unifies multimodal understanding and generation in autoregressive models.
• Context Unrolling in Omni Models — Unified multimodal training enables "Context Unrolling" for enhanced cross-modal reasoning and performance.
• Vision as Unified Multimodal Generation — Formulating computer vision as unified multimodal generation enables a single model to handle diverse visual tasks via text and image outputs.
• What is Multimodal AI? How LLMs Process Text, Images, and More — Native multimodality uses a shared vector space for unified processing of diverse data types, enabling richer AI reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.