PercepCap: Video Captioner with Structured Spatio-Temporal Perception

2026-05-07 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, long

Summary

PercepCap is a novel perception-aware video captioning framework designed to address limitations in existing Multi-modal Large Language Models (MLLMs) that lack explicit perceptual evidence. It employs a "perceive–describe" generation chain, where the model first produces a structured spatio-temporal perception trace, including object trajectories and temporal events, before generating the final caption. This approach makes underlying perceptual errors identifiable. PercepCap is trained using a two-stage strategy: Perceive-then-Describe Supervised Fine-tuning (PD-SFT) adapts the model to this new chain, followed by Perception-Grounded Reinforcement Learning (PG-RL) which optimizes both the perception trace and caption quality with joint rewards. A key component is Caption-Anchored Perception Data Construction, which creates aligned training data by grounding extracted caption content back into video. Evaluated on benchmarks like DREAM-1K, CaReBench, and MotionBench, PercepCap consistently outperforms the Qwen3-VL baseline, demonstrating leading caption quality.

Key takeaway

For Machine Learning Engineers developing advanced video captioning systems, PercepCap offers a robust framework to enhance model accuracy and interpretability. You should consider adopting a "perceive–describe" architecture to explicitly ground captions in spatio-temporal evidence. This approach, supported by two-stage fine-tuning and caption-anchored data, can significantly improve object and event recognition, leading to more faithful and auditable video descriptions in your applications.

Key insights

PercepCap explicitly separates spatio-temporal perception from caption generation for improved video understanding.

Principles

• Explicit perceptual evidence improves captioning accuracy.
• Decomposed rewards enhance perception and caption alignment.
• Caption-anchored data ensures perception-caption consistency.

Method

PercepCap uses a perceive–describe chain: generate structured spatio-temporal trace (objects, events), then caption. Trained via PD-SFT and PG-RL with caption-anchored data.

In practice

• Use structured perception traces for video search.
• Enhance accessibility descriptions with grounded captions.
• Audit video datasets for object/event consistency.

Topics

• PercepCap
• Video Captioning
• Multi-modal LLMs
• Spatio-Temporal Perception
• Reinforcement Learning
• Object Tracking

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• PercepCap: Video Captioner with Structured Spatio-Temporal Perception — Explicitly generating spatio-temporal perception traces before captioning significantly improves video understanding and error identification.
• PercepCap: Video Captioner with Structured Spatio-Temporal Perception — PercepCap explicitly models spatio-temporal perception before caption generation, improving accuracy and error identification.
• ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning — ProCap iteratively refines video captions by prioritizing prominent objects, significantly reducing hallucination and increasing completeness without retraining models.
• VEGAS: Human-Aligned Video Caption Evaluation via Gaze — VEGAS uses test-time gaze data as a training-free, cross-modal metric to select video captions that align with individual viewer attention, improving retrieval.
• Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding — F2G improves video temporal grounding by decoupling event identification from precise boundary measurement using citable evidence.
• PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models — Multimodal diffusion language models can achieve parallel, efficient multi-region perception by leveraging non-autoregressive generation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.