PercepCap: Video Captioner with Structured Spatio-Temporal Perception

2026-07-22 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

PercepCap is a novel perception-aware video captioning framework designed to improve spatio-temporal understanding by making perceptual evidence explicit. Unlike existing MLLMs that directly generate captions, PercepCap employs a "perceive-describe" generation chain. This process first produces a spatio-temporal perception trace, detailing object trajectories and temporal events, before generating the final caption conditioned on this evidence. To facilitate this, the framework utilizes a two-stage training strategy: Perceive-then-Describe Supervised Fine-tuning adapts the model, followed by Perception-Grounded Reinforcement Learning that optimizes both the perception trace and caption quality using joint rewards. A key component is Caption-Anchored Perception Data Construction, which creates training data by grounding objects and events extracted from initial captions back into video with precise boxes and timestamps. PercepCap consistently improves upon the Qwen3-VL baseline across direct caption and caption-to-QA evaluations, demonstrating leading caption quality.

Key takeaway

For Machine Learning Engineers developing video captioning systems, PercepCap offers a robust approach to enhance accuracy and debug errors. You should consider implementing a perceive-describe generation chain to explicitly model spatio-temporal perception, rather than relying solely on direct caption generation. This method provides clearer insights into underlying perceptual mistakes, allowing you to refine model training and improve overall caption quality, especially when using models like Qwen3-VL.

Key insights

PercepCap explicitly models spatio-temporal perception before caption generation, improving accuracy and error identification.

Principles

• Explicit perceptual evidence improves video captioning.
• Grounding extracted entities enhances training data quality.
• Joint rewards optimize both perception and caption output.

Method

PercepCap uses a perceive-describe chain: generate spatio-temporal trace, then caption. It's trained with two stages: SFT and Perception-Grounded RL, supported by Caption-Anchored Perception Data Construction.

In practice

• Use explicit perception traces for debugging MLLM errors.
• Apply two-stage SFT and RL for complex generation tasks.
• Construct perception data by grounding caption entities.

Topics

• Video Captioning
• Spatio-Temporal Perception
• Multimodal LLMs
• Reinforcement Learning
• Supervised Fine-tuning
• Qwen3-VL

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• PercepCap: Video Captioner with Structured Spatio-Temporal Perception — PercepCap explicitly separates spatio-temporal perception from caption generation for improved video understanding.
• PercepCap: Video Captioner with Structured Spatio-Temporal Perception — Explicitly generating spatio-temporal perception traces before captioning significantly improves video understanding and error identification.
• From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas — Intermediate machine translation fine-tuning significantly improves vision-language models for Indigenous language image captioning.
• GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models — Naively using VLM-generated captions can degrade accuracy due to anchoring effects and asymmetric error types.
• CineCap: Structured Reasoning with Spatio-Temporal Anchors for Cinematographic Video Captioning — CineCap uses structured reasoning and reinforcement learning to generate accurate cinematographic video captions.
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding — Video generation models inherently learn robust 3D structural priors and physical laws for scene understanding.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.