Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images

2026-07-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study evaluated 25 Vision Transformers (ViTs), trained with both supervised and self-supervised objectives, to determine if they learn Gestalt-like figure-ground organization cues from natural images. Researchers fitted linear probes to predict figure-ground assignment from intermediate patch representations, utilizing both natural images and controlled artificial stimuli designed to isolate individual cues. The findings indicate that ViTs robustly encode surroundedness and convexity, with probes trained on natural images demonstrating zero-shot generalization to artificial stimuli across several models. However, symmetry encoding showed mixed results, being present for uniformly colored regions but not for textured ones. These results suggest that Gestalt-like figure-ground cues can emerge from natural scene statistics, establishing ViTs as a compelling model system for investigating the computational mechanisms of perceptual organization.

Key takeaway

For research scientists investigating visual perception in AI, this study suggests Vision Transformers offer a powerful model system. You should explore ViT intermediate representations to understand how specific Gestalt cues like surroundedness and convexity emerge from natural image training. Consider utilizing ViTs to develop more robust and human-aligned figure-ground segmentation algorithms, particularly by analyzing their generalization capabilities across diverse visual contexts.

Key insights

Vision Transformers acquire Gestalt-like figure-ground organization cues directly from natural image statistics.

Principles

• ViTs robustly encode surroundedness and convexity.
• Training on natural images enables zero-shot generalization.
• Symmetry encoding varies with region texture.

Method

Linear probes predict figure-ground assignment from ViT intermediate patch representations, using both natural images and controlled artificial stimuli.

In practice

• Utilize ViTs for perceptual organization studies.
• Examine ViT layers for specific visual cue encoding.
• Evaluate model generalization with varied stimuli.

Topics

• Vision Transformers
• Figure-Ground Organization
• Gestalt Cues
• Perceptual Organization
• Self-Supervised Learning
• Computer Vision

Code references

• mtangemann/mlvbench

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perception — Vision Transformers better align with human texture perception than CNNs, driven by architecture.
• GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks — GH-ESD reformulates error slice discovery for instance-level vision tasks using grounded hypothesis generation and statistical verification.
• Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation — ViTC-UNet combines frozen ViT global priors with UNet local inductive biases for efficient, high-precision biomedical semantic segmentation.
• Masked Autoencoder for Image Reconstruction using Vision Transformers (PyTorch) — Masked Autoencoders with Vision Transformers learn robust image representations by reconstructing heavily masked image patches.
• Unsupervised Semantic Segmentation Facilitates Model Understanding — Visualizing unsupervised semantic segmentation reveals consistent, interpretable behaviors in self-supervised vision transformers.
• Crafting the Eyes for Thinking Machines: Rewiring the Retina- The Anatomy of ViTStruct — ViTStruct physically separates visual information into distinct streams to enable more precise reasoning in Vision Transformers.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.