Zero-Shot Captioning for Cultural Heritage: Automated Image Analysis of Traditional Indonesian Clothing

2026-06-11 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

Custom ZeroCLIP is a novel retrieval-augmented vision-language framework designed for zero-shot captioning of traditional Indonesian garments. This framework utilizes a dataset of 3,800 expert-annotated images spanning all 38 Indonesian provinces. It employs a province-level inductive zero-shot protocol, training on 24 seen provinces, validating on 6 seen, and evaluating on 8 unseen provinces. The architecture integrates a frozen CLIP ViT-B/32 image encoder, CLIP and BERT text encoders, and an LSTM caption decoder. During inference, the model retrieves captions solely from training provinces, ensuring no unseen-province data is used. Custom ZeroCLIP achieved a CLIPScore of 0.8536, BLEU-4 of 0.3342, and METEOR of 0.4859, surpassing existing baselines. Ablation studies confirmed retrieval's role in enhancing cultural vocabulary recovery, yielding a 19.3% METEOR gain, with human evaluation validating its cultural accuracy and fluency. The dataset is publicly available.

Key takeaway

For Computer Vision Engineers developing AI for cultural heritage, Custom ZeroCLIP demonstrates a robust approach to zero-shot captioning. You should consider integrating retrieval-augmented vision-language frameworks to improve cultural vocabulary recovery and accuracy in low-resource settings. This method, particularly with inductive zero-shot protocols, allows your models to generalize effectively to unseen regional variations, enhancing the utility of automated image analysis for diverse cultural artifacts.

Key insights

Retrieval-augmented vision-language models effectively generate culturally accurate captions for low-resource heritage data.

Principles

• Inductive zero-shot protocols enable generalization to unseen domains.
• Retrieval augmentation significantly boosts cultural vocabulary recovery.
• Combining multiple encoders enhances domain adaptation.

Method

Custom ZeroCLIP combines a frozen CLIP ViT-B/32 image encoder, CLIP and BERT text encoders, and an LSTM caption decoder, using retrieval from seen-province captions during inference for unseen provinces.

In practice

• Apply retrieval-augmented captioning for niche cultural datasets.
• Use province-level zero-shot for diverse regional variations.
• Integrate multiple pre-trained encoders for robust domain adaptation.

Topics

• Zero-Shot Captioning
• Cultural Heritage AI
• Vision-Language Models
• CLIP
• Retrieval Augmentation
• Indonesian Garments

Code references

• AnugrahAidinYotolembah/Traditional-Indonesian-Clothing-Captioning-Dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• OpenAI CLIP: The Model That Learnt Zero-Shot Image Recognition Using Text — CLIP connects vision and language through contrastive learning on internet-scale image-text pairs, enabling zero-shot image classification.
• VietFashion: Benchmarking Sketch-Text Composed Image Retrieval for Cultural Outfits — Cultural garment retrieval requires benchmarks that capture subtle structural and semantic details.
• Boosting Brain-to-Image Decoding with TRIBE v2 Data Augmentation — Synthetic fMRI data from TRIBE v2 significantly boosts brain-to-image decoding accuracy, enabling zero-shot capabilities.
• Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets — Genuine zero-shot image retrieval benchmarks require novel, consistent data and robust evaluation to counter inflated performance from pre-trained models.
• ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China — VLMs demonstrate strong visual recognition but struggle with culturally-grounded reasoning on heritage sites.
• CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling — Random sampling and alternating batching are more effective than explicit class balancing for 3D medical vision-language models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.