MOSAIC: Adaptive Inter-layer Composition for Efficient Heterogeneous Vision-Language Models

2026-07-10 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

MOSAIC (Multi-Objective Search for Adaptive Inter-layer Composition) is a hardware-aware search method designed to automatically transform homogeneous Vision-Language Models (VLMs) into optimized heterogeneous architectures. It addresses the limitations of handcrafted static mixing patterns by integrating diverse efficiency mechanisms, including linear, sparse, and low-rank operators, into a unified search space. The method formulates selection as a multi-objective Mixed Integer Programming (MIP) problem to maximize performance under strict hardware latency constraints. A two-stage parameter recovery process, involving global off-policy distillation and dual-teacher on-policy distillation with a 235B oracle and 4B teacher, mitigates performance degradation. Validated through MOSAIC-4B, derived from Qwen3-VL-4B-Instruct, the approach matches baseline performance across benchmarks while requiring less than 2% of original training cost and achieving 1.76x prefilling and 2.54x decoding speedups.

Key takeaway

For Machine Learning Engineers deploying Vision-Language Models, MOSAIC offers a systematic approach to significantly enhance inference efficiency without compromising performance or incurring high retraining costs. If your team faces hardware latency constraints with homogeneous VLM architectures, consider exploring hardware-aware architectural search methods like MOSAIC, coupled with multi-stage distillation, to achieve substantial speedups in prefilling and decoding.

Key insights

MOSAIC adaptively optimizes VLM architectures for hardware efficiency using multi-objective search and distillation.

Principles

• Heterogeneous VLM structures improve performance and latency.
• Hardware-aware search optimizes VLM layer composition.
• Distillation stabilizes performance during structural transitions.

Method

MOSAIC employs multi-objective Mixed Integer Programming to select efficient operators (linear, sparse, low-rank) for VLM layers, followed by a two-stage distillation process for parameter recovery.

In practice

• Apply MIP for VLM architecture optimization.
• Use dual-teacher distillation for VLM fine-tuning.
• Integrate diverse operators like linear/sparse attention.

Topics

• Vision-Language Models
• Heterogeneous Architectures
• Model Optimization
• Inference Efficiency
• Neural Architecture Search
• Knowledge Distillation

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models — VisPCO optimizes visual token pruning in VLMs by formulating it as a Pareto configuration problem for optimal efficiency.
• MOSAIC: Modular Orchestration for Structured Agentic Intelligence and Composition — MOSAIC structures LLM-based agentic model selection through blueprints, improving automated data science performance and traceability.
• GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning — GRASS adaptively samples LLM layers for fine-tuning based on gradient norms, significantly reducing memory while boosting performance.
• The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models — Multi-teacher prompt distillation significantly improves VLM student performance, particularly under domain shift, through complementary teacher supervision.
• Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems — MOSAIC optimizes data selection for physical AI by iteratively adding data from domains that maximize metric changes.
• SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs — SAMPLe enhances VLM prompt generalization by optimizing for flatter, lower-loss minima through dynamic sharpness-aware gradient adjustments.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.