BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model

2026-05-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

BEiTScore introduces a new reference-free image captioning evaluation metric, leveraging a lightweight cross-encoder model initialized from a Vision Question Answering (VQA) checkpoint. This metric addresses limitations of existing methods, such as the high computational costs of Large Language Models (LLMs) or the token limits and "bag-of-words" issues of CLIP-based encoders. BEiTScore's training scheme incorporates a carefully assembled data mixture, featuring adversarial LLM-based data augmentations to enhance sensitivity to fine-grained visual-linguistic errors. The paper also introduces LongCapVLCP, a new benchmark designed to assess detailed captioning evaluation across diverse scenarios, including long-form narratives and in-image text recognition. Experimental results demonstrate that BEiTScore achieves state-of-the-art performance, matching or surpassing LLM-based metrics like EXPERT, FLEUR, and VQAScore, while being 20x to 60x lighter and 30x to 100x faster at inference time.

Key takeaway

For machine learning engineers evaluating vision-language models, BEiTScore provides a highly efficient and accurate reference-free metric. You can achieve state-of-the-art human correlation and fine-grained error detection, even for long captions and scene text, without the computational overhead of large language models. Consider integrating BEiTScore to accelerate your model development and large-scale benchmarking efforts, especially when evaluating models generating complex or lengthy descriptions.

Key insights

BEiTScore offers an efficient, reference-free image captioning evaluation metric, outperforming LLMs on complex tasks.

Principles

• Smaller models can match LLM performance with targeted training data exposure.
• Adversarial LLM-driven data augmentation enhances sensitivity to fine-grained visual-linguistic errors.
• Multi-stage training improves fine-grained perception and compositional understanding.

Method

BEiTScore uses a two-stage training: pairwise contrastive learning with LLM-augmented data for fine-grained errors, then fine-tuning on human judgment datasets, warm-started from a VQA checkpoint.

In practice

• Use BEiTScore for large-scale VLM benchmarking.
• Apply BEiTScore for quality-aware caption decoding.
• Integrate BEiTScore for reward-based caption guidance.

Topics

• Image Captioning Evaluation
• Vision-Language Models
• Cross-Encoder Models
• BEiT-3
• LLM Data Augmentation
• Reference-Free Metrics
• LongCapVLCP Benchmark

Code references

• microsoft/unilm

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models — Naively using VLM-generated captions can degrade accuracy due to anchoring effects and asymmetric error types.
• Multilingual Coreference Resolution via Cycle-Consistent Machine Translation — Coreference resolution in low-resource languages can be significantly improved by cycle-consistent machine translation data augmentation.
• CIAN: Multi-Stage Framework for Event-Enriched Image Captioning via Retrieval-Augmented Generation — CIAN enriches image captions with external event context using retrieval-augmented generation and linguistic refinement.
• System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5 — Domain-specific datasets and LoRA fine-tuning significantly enhance LLM performance in specialized tasks like classical poetry analysis.
• Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality — MACCO improves VLM compositional understanding by cross-modal masked concept reconstruction and auxiliary alignment objectives.
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis — Current LVLMs struggle with fine-grained image recognition due to intertwined bottlenecks in visual representations, semantic grounding, and knowledge.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.