Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

MiRA (Marginal-induced Attention Redistribution) is a novel plug-in framework designed to enhance Vision Transformer (ViT) backbones for understanding subtle facial expressions in videos. It addresses the limitation of standard ViT models, which often prioritize dominant global motions over fine-grained facial dynamics. MiRA achieves this by deriving frame-level confidence and intra-frame concentration statistics from self-attention maps, then redistributing attention towards localized facial cues without adding trainable parameters. The framework offers two modes: an "exact mode" for principled post-softmax redistribution and a "flashLite mode," an efficient pre-softmax approximation compatible with FlashAttention kernels. Experimental results demonstrate MiRA's consistent improvements on challenging Facial Expression Recognition (FER) benchmarks, including DFEW, MAFW, and FERV39k, outperforming several general video models. The flashLite mode further provides a 20% latency reduction and 26% throughput gain on an H100 GPU, making it highly efficient.

Key takeaway

For Machine Learning Engineers developing video-based facial expression recognition systems, integrating MiRA into your Vision Transformer backbones can significantly improve performance. This plug-in module enhances sensitivity to subtle facial dynamics by intelligently redistributing attention, outperforming existing general video models. Consider deploying the flashLite mode for its computational efficiency, offering a 20% latency reduction and 26% throughput gain on H100 GPUs, especially when scaling to larger models or real-world applications without task-specific preprocessing.

Key insights

Reweighting ViT attention based on frame-level confidence and intra-frame concentration improves subtle facial dynamics understanding.

Principles

• Frame-level confidence identifies salient frames.
• Intra-frame concentration promotes localized attention.
• Pre-softmax approximation can preserve effectiveness.

Method

MiRA computes frame-level confidence and intra-frame concentration from attention maps (exact) or key energies (flashLite), combines them into a marginal importance score, and uses this to redistribute attention via scaling factors or logit biases.

In practice

• Integrate MiRA as a plug-in module into ViT backbones.
• Use flashLite mode for efficiency with FlashAttention.
• Apply full-depth reweighting for best performance.

Topics

• Facial Expression Recognition
• Video Transformers
• Self-Attention Mechanisms
• FlashAttention
• MiRA Framework
• Spatio-Temporal Dynamics

Code references

• ysrinria/MiRA

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Facial-Expression-Aware Prompting for Empathetic LLM Tutoring — Integrating structured facial expression data via prompt engineering significantly boosts LLM tutor empathy without retraining.
• DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning — Integrating infrared and RGB data via localized cross-attention significantly boosts MLLM robustness in degraded visual conditions.
• DnA: Denoising Attention for Visual Tasks — DnA improves visual attention by separating relevant and irrelevant features into distinct subspaces.
• EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units — EmoZone-Talker disentangles speech-driven and explicit facial expressions for precise 3DGS talking head control.
• Scaling View Synthesis Transformers — Encoder-decoder architectures can be compute-optimal for view synthesis transformers with proper design and training strategies.
• HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers — Unifying image and video tokenization in a single ViT with specific architectural choices significantly enhances multimodal model performance and editing consistency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.