PercepCap: Video Captioner with Structured Spatio-Temporal Perception

2026-07-22 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

PercepCap is a novel perception-aware video captioning framework designed to address limitations in existing Multimodal Large Language Models (MLLMs) that obscure underlying spatio-temporal perception errors. It operates on a "perceive-describe" generation chain, first producing an explicit spatio-temporal perception trace, including object trajectories and temporal events, and then generating the final caption conditioned on this evidence. PercepCap employs a two-stage training strategy: Perceive-then-Describe Supervised Fine-tuning adapts the model, followed by Perception-Grounded Reinforcement Learning to optimize both the perception trace and caption quality. A Caption-Anchored Perception Data Construction pipeline supports this by generating caption-aligned perception ground truth. This approach consistently improves upon the Qwen3-VL baseline, demonstrating leading caption quality in direct caption and caption-to-QA evaluations.

Key takeaway

For Machine Learning Engineers developing video captioning systems, PercepCap's approach offers a clear path to improving model interpretability and accuracy. By explicitly generating spatio-temporal perception traces before captioning, you can more easily identify and correct underlying perceptual errors. Consider implementing a "perceive-describe" chain and two-stage training to enhance your model's understanding and caption quality, moving beyond opaque end-to-end generation.

Key insights

Explicitly generating spatio-temporal perception traces before captioning significantly improves video understanding and error identification.

Principles

• Exposing perceptual evidence aids error diagnosis.
• Conditioning captions on explicit traces enhances quality.
• Joint rewards optimize perception and captioning.

Method

PercepCap uses a two-stage training: Supervised Fine-tuning for perceive-describe adaptation, then Perception-Grounded Reinforcement Learning with joint rewards. Caption-Anchored Perception Data Construction provides aligned ground truth.

In practice

• Generate object trajectories and temporal events.
• Condition caption generation on these explicit traces.
• Construct caption-aligned perception data.

Topics

• Video Captioning
• Spatio-Temporal Perception
• Multimodal Large Language Models
• Reinforcement Learning
• Object Trajectories
• Event Grounding
• Qwen3-VL

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• PercepCap: Video Captioner with Structured Spatio-Temporal Perception — PercepCap explicitly separates spatio-temporal perception from caption generation for improved video understanding.
• PercepCap: Video Captioner with Structured Spatio-Temporal Perception — PercepCap explicitly models spatio-temporal perception before caption generation, improving accuracy and error identification.
• ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning — ProCap iteratively refines video captions by prioritizing prominent objects, significantly reducing hallucination and increasing completeness without retraining models.
• PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models — Multimodal diffusion language models can achieve parallel, efficient multi-region perception by leveraging non-autoregressive generation.
• Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events — Current video MLLMs struggle with momentary visual events due to sampling, compression, and aggregation issues, lacking temporal fidelity.
• VEGAS: Human-Aligned Video Caption Evaluation via Gaze — VEGAS uses test-time gaze data as a training-free, cross-modal metric to select video captions that align with individual viewer attention, improving retrieval.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.