LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

LAViFiT is an end-to-end latent-action-guided framework designed for vision-language fine-tuning in surgical interaction recognition. It addresses challenges in adapting pretrained vision-language models and encoders to fine-grained surgical tasks, specifically the issues of noisy representations from frozen encoders and limited spatial precision from full fine-tuning. The framework integrates an inverse dynamics model to capture action-induced visual changes and a forward world model to guide the encoder towards action-relevant regions. A novel Patch-level SIG Regularizer prevents local feature collapse without requiring additional supervision like bounding boxes. Experiments on CholecT50 and ProstaTD datasets demonstrate that LAViFiT improves recognition and image-text alignment across various encoders, including ViT-L/14, DINOv2-L/14, and SurgeNet-L/14. The SurgeNet-L model achieved the best overall results, with 18.50 mAP on CholecT50 and 27.51 mAP on ProstaTD. The method also yields stronger grounding in instrument-tissue interaction regions and more spatially coherent features. The source code is publicly available.

Key takeaway

For AI Scientists and Machine Learning Engineers developing surgical AI systems, if you are struggling with adapting general-domain vision models to fine-grained instrument-tissue interaction recognition, you should consider implementing the LAViFiT framework. This method significantly improves recognition performance and feature grounding by explicitly guiding your vision encoder to focus on action-relevant regions. You can achieve more spatially coherent and interpretable features, but be aware that world model objectives with SIG-style regularization can be sensitive to hyperparameter choices.

Key insights

Latent-action modeling with patch-level regularization precisely fine-tunes vision encoders for surgical instrument-tissue interaction recognition.

Principles

• Latent action models guide encoders to action-relevant regions.
• Low-dimensional action bottlenecks compress salient transitions.
• Patch-level regularization prevents local feature collapse.

Method

LAViFiT jointly trains a vision encoder, Delta-IDM, DiT-style FWM, clip aggregator, and text encoder. It uses classification, forward prediction, and state/patch-level SIG regularization losses.

In practice

• Improve surgical triplet recognition via latent-action guidance.
• Enhance feature grounding with patch-level SIG regularization.
• Optimize action dimension for specific encoder and task.

Topics

• Surgical AI
• Vision-Language Models
• Latent Action Models
• Surgical Interaction Recognition
• Feature Grounding
• Patch-level Regularization

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition — LAViFiT uses latent-action guidance and regularization to fine-tune VLMs for precise surgical interaction recognition.
• SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics — SurgVista improves surgical world models by addressing interaction incoherence and temporal drift for better long-horizon predictions.
• A Unified Tokenization Framework for Pain Recognition using Heterogeneous 3D Modalities — A unified tokenization framework processes diverse 3D modalities for pain recognition, achieving high performance and real-time efficiency.
• Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions — FAROS interpolates sparse surgical annotations into dense, temporally consistent labels using optical flow, balancing multi-task learning.
• OR-Action: Multi-Role Video Understanding with Fine-Grained Actions — Fine-grained OR video understanding requires explicit temporal modeling beyond scene graphs, benefiting from vision-only temporal models and multi-view alignment.
• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition — Multi-agent VLM deliberation with heterogeneous specialists improves fine-grained egocentric action recognition by decorrelating model priors.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.