ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory

2026-06-25 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

ProtoKV is a novel constant-footprint memory system designed for Streaming Video Understanding (SVU) tasks, specifically addressing the challenge of delayed queries. SVU systems must process continuous visual token streams and answer asynchronous queries under strict GPU memory and latency constraints. The core problem is that critical visual cues can appear briefly but be evicted or diluted from bounded memory before a delayed query arrives. ProtoKV tackles this by maintaining an exact near-window Key-Value (KV) cache and aggregating older content into a fixed-capacity semantic-spatial prototype bank with residual statistics. This approach represents far history as a summary state instead of retaining individual token instances. At query time, prototypes are exposed via a bounded pseudo-token interface compatible with standard attention mechanisms. ProtoKV demonstrates accuracy improvements of up to 12.5 points over token-retention baselines on SVU benchmarks in long-delay scenarios, with gains increasing as query delay extends.

Key takeaway

For Machine Learning Engineers developing streaming video understanding systems, especially those facing delayed query challenges, ProtoKV offers a significant architectural improvement. You should consider integrating its constant-footprint summary-state memory to mitigate the risk of critical cue eviction. This approach can boost accuracy by up to 12.5 points in long-delay scenarios, ensuring your models retain vital historical context without exceeding GPU memory budgets. Evaluate ProtoKV's prototype-based aggregation for enhanced performance in real-time applications.

Key insights

ProtoKV uses a summary-state memory with a prototype bank to efficiently handle delayed queries in streaming video understanding.

Principles

• Constant-footprint memory is crucial for SVU.
• Aggregate far history into summary states.
• Semantic-spatial prototypes improve recall.

Method

ProtoKV combines an exact near-window KV cache with a fixed-capacity semantic-spatial prototype bank for older content. Prototypes are exposed via a bounded pseudo-token interface compatible with standard attention at query time.

In practice

• Apply to streaming video analytics.
• Improve accuracy in long-delay SVU.
• Integrate with standard attention models.

Topics

• Streaming Video Understanding
• Delayed Query
• Memory Management
• Key-Value Cache
• Prototype Learning
• Computer Vision

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams — LiveStarPro enables proactive, real-time video understanding for long streams by integrating hierarchical memory and efficient decoding.
• WorldKV: Efficient World Memory with World Retrieval and Compression — WorldKV uses retrieval and compression to maintain persistent world memory in video diffusion models efficiently.
• Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly — Current LVLMs significantly struggle with fine-grained spatio-temporal reasoning and object interaction in complex video scenarios.
• What Should a Streaming Video Model Remember? — SelectStream selectively allocates latent memory for streaming video understanding, balancing current perception with historical context.
• Reason-Then-Retrieve for CoVR-R with Structured Edit Prompts and Dense-Sparse Fusion — Structured prompts and dense-sparse fusion are crucial for effective reason-aware composed video retrieval.
• Microsoft Research's Mirage gives video generation a persistent spatial memory that doesn't forget what's around the corner — Mirage uses latent spatial memory to achieve persistent spatial consistency and efficiency in video world models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.