zzunlp at ClinicalSkillQA: Perceive-and-Plan with Decomposed In-Context Learning and Saliency-Guided Perception for Clinical Skill Keyframe Reordering

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Clinical Care & Medical Practice · Depth: Expert, quick

Summary

The zzunlp team introduces Perceive-and-Plan, a decomposed in-context learning paradigm designed to enhance Multimodal Large Language Models' (MLLMs) continuous perception in procedural clinical workflows, specifically for clinical skill keyframe reordering. This method operates without parameter updates and separates visual perception from temporal planning into two stages. The first stage employs structured visual perception with saliency-guided Picture-in-Picture (PiP) composition, magnifying critical regions like the head and chest as color-coded insets. The second stage involves temporal reasoning through chain-style self-verification, utilizing fresh conversation resets and visual-evidence anchoring via BLS Rules R1-R11. The system achieved an overall score of 71.43 (2nd place, ClinSkill QA 2026), with 0.86 pairwise accuracy and 1.0 rationale coverage, demonstrating that structured prompting with visual saliency guidance measurably improves MLLMs' procedural understanding.

Key takeaway

For AI Scientists and Machine Learning Engineers developing MLLMs for medical procedural understanding, adopting a decomposed in-context learning approach like Perceive-and-Plan can significantly enhance performance. You should explore separating visual perception with saliency-guided Picture-in-Picture from temporal reasoning using chain-style self-verification to achieve higher accuracy and rationale coverage in complex clinical skill reordering tasks. This strategy improves MLLM procedural understanding without parameter updates.

Key insights

Decomposed in-context learning with saliency-guided perception improves MLLM procedural understanding for clinical keyframe reordering.

Principles

• Separate visual perception from temporal planning.
• Saliency-guided visual composition enhances MLLM focus.
• Chain-style self-verification improves temporal reasoning.

Method

The method involves structured visual perception using saliency-guided PiP to magnify critical regions, followed by temporal reasoning with chain-style self-verification and visual-evidence anchoring (BLS Rules R1-R11).

In practice

• Use PiP composition to highlight critical visual areas.
• Implement conversation reset for self-verification in temporal tasks.
• Anchor reasoning with specific visual evidence rules.

Topics

• Multimodal Large Language Models
• Clinical Skill QA
• Keyframe Reordering
• In-Context Learning
• Saliency-Guided Perception
• Procedural Understanding

Code references

• NanceTide/clinskillqa-perceive-and-plan

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA — Chain-of-Procedure (CoP) enhances VLM performance in visual procedural QA by addressing retrieval and granularity issues.
• Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation — Self-distillation from regional crops to full images improves MLLM fine-grained visual understanding.
• Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge — Bi-level prompt optimization improves multimodal LLM judge alignment by converting images to text and co-optimizing judge and I2T prompts.
• MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models — MedSIGHT unifies medical visual comprehension and pixel-level segmentation via region-centric tokens and a modality-aware codebook.
• Thinking Like a Botanist: Challenging Multimodal Language Models with Intent-Driven Chain-of-Inquiry — Structured, intent-driven inquiry significantly enhances MLLM diagnostic accuracy and reasoning efficiency in botanical pathology.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.