Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

2026-06-24 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A new audit, Facet-Probe, investigates order sensitivity in 18 frontier and open-weight multimodal large language models (MLLMs) across five facets: option, evidence-chunk, document-rank, image-set, and mixed-modality ordering. Standard benchmarks often overlook whether shuffling order-irrelevant inputs changes MLLM answers, a critical reliability property. The study employs a Bayesian item-response model to distinguish ordering noise from per-facet bias and uses a same-ordering control to estimate the decoder-stochastic floor. Results indicate none of the 18 MLLMs are order-invariant, with screened per-facet panel-mean flip rates ranging from 24% to 50%. Even the best model exhibits flips on 13.4% of trials, demonstrating a substantial ordering excess over decoder noise. Training-free prompt changes tested on Gemini showed modality-conditional effects, not transferring from text to visual reasoning, suggesting prompt-level mitigation alone is insufficient for general order robustness. The authors propose cross-ordering flip rate as a standard reporting axis for MLLMs.

Key takeaway

For MLOps Engineers deploying multimodal LLMs, you must account for significant order sensitivity. Your models, even top-performing ones, will likely produce different answers when input elements are merely reordered, impacting reliability. Do not rely solely on prompt engineering for robustness, as these changes are often modality-specific. Integrate cross-ordering flip rate into your evaluation pipelines to ensure consistent model behavior under varied input presentations.

Key insights

Multimodal LLMs exhibit significant order sensitivity, with current prompt-level mitigations proving insufficient for general robustness.

Principles

• MLLM reliability requires auditing order-irrelevant input shuffling.
• Capability does not eliminate order sensitivity in MLLMs.
• Prompt-level mitigations are modality-conditional and non-transferable.

Method

Facet-Probe audits MLLMs across five ordering facets, using a Bayesian item-response model to separate noise from bias and a same-ordering control for decoder-stochastic floor estimation.

In practice

• Audit MLLMs for order sensitivity using diverse input permutations.
• Report cross-ordering flip rates as a standard MLLM metric.

Topics

• Multimodal LLMs
• Order Sensitivity
• AI Evaluation
• Model Robustness
• Facet-Probe
• Prompt Engineering

Code references

• NKU-HLT/DIFFA

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations — Large Language Models, including medical-specific variants, are critically fragile to prompt variations in healthcare.
• Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation — Soft-prompt tuning efficiently reveals true LLM knowledge by overcoming format-following limitations in benchmarks.
• On the Adversarial Robustness of Multimodal LLM Judges — MLLM judges are vulnerable to adversarial attacks, necessitating robust evaluation frameworks and new attack methods like MGSIA.
• Month in 4 Papers (April 2026) — LLM accuracy can unexpectedly improve with "rude" prompts compared to "polite" ones.
• Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads — MLLMs exhibit functional sparsity via Context-aware Retrieval (CoRe) heads, crucial for efficient cross-modal information extraction.
• Is Opus Dumb Today? — LLM performance degradation often results from unverified context rot, not inherent model failure.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.