Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly

2025-11-13 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Flat-Pack Bench is a novel benchmark designed to evaluate Large Vision-Language Models (LVLMs) on fine-grained spatio-temporal understanding, specifically within furniture assembly tasks. It addresses a gap in existing benchmarks that primarily focus on coarse-grained video understanding. The benchmark features nuanced tasks, including temporal ordering of assembly actions, temporal localization of assembly state, understanding part mating, and tracking, using multiple-choice questions paired with visual prompts. Experiments reveal that state-of-the-art LVLMs, such as OpenAI's GPT-5 and Google's Gemini 2.5/3.1 Pro, struggle significantly, achieving accuracies around 38% and 33% respectively, far below human performance of 94.18%. Among open models, InternVL3-78B performed best at 41.03%.

Key takeaway

For AI scientists and machine learning engineers developing video understanding models, this benchmark highlights critical limitations in spatio-temporal reasoning. You should prioritize improving models' ability to track multiple similar objects, understand physical interactions, and effectively leverage temporal context from long videos. Current linguistic prompting or agentic decomposition approaches do not significantly bridge this performance gap, indicating a need for fundamental architectural or training data advancements.

Key insights

Current LVLMs significantly struggle with fine-grained spatio-temporal reasoning and object interaction in complex video scenarios.

Principles

• LVLMs underutilize temporal information in videos.
• Effective object tracking is a major bottleneck for LVLMs.
• Physical interaction understanding is limited in current models.

Method

Flat-Pack Bench augments the IKEA-Manuals-at-Work dataset with manually annotated part segmentations and curated multiple-choice questions across four spatio-temporal understanding tasks.

In practice

• Evaluate models on temporal ordering of events.
• Test models on part mating and contact detection.
• Use visual prompts with high-contrast labels.

Topics

• Large Vision-Language Models
• Spatio-Temporal Reasoning
• Video Understanding
• Benchmark
• Object Tracking
• Furniture Assembly

Code references

• justachetan/flat-pack-bench

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis — VGenST-Bench shifts spatio-temporal reasoning evaluation from passive curation to active, controlled video synthesis for MLLMs.
• Watch, Remember, Reason: Human-View Video Understanding with MLLMs — MLLM video understanding benefits from a human-like "watch, remember, reason" functional decomposition.
• Can you *really* train AI to "get" videos just by showing it a million of them? — Current video models excel at generation but lack systematic evaluation for genuine spatiotemporal reasoning.
• Mind the Gap: Disentangling Performance Bottlenecks in Video Instance Segmentation — Tracking instability, not classification or representation, is the primary bottleneck for online Video Instance Segmentation methods.
• Masked Diffusion Vision-Language Models for Temporal Action Localization — Masked diffusion vision-language models enable joint, iterative refinement of temporal boundaries and semantic content in untrimmed videos.
• Temporal Preservation over Processing: Diagnosing and Designing Spatiotemporal Single-Stage Video Detectors — Standard mAP metrics hide whether video detectors truly use temporal context.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.