VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Environmental Science & Earth Systems · Depth: Expert, medium

Summary

The Vision-Language Caption Enhancer (VLCE) is a framework designed to improve image descriptions for post-disaster satellite and UAV imagery, addressing the limitations of general-purpose vision-language models like LLaVA and QwenVL which often lack domain-specific vocabulary. VLCE operates in two stages: first, a baseline VLM generates an initial caption based on YOLOv8 object detections; second, a knowledge-enriched sequential model, either a CNN-LSTM or a hierarchical cross-modal Transformer, refines this caption. The refinement process integrates external semantic knowledge from ConceptNet and WordNet, augmenting the vocabulary with 1,566 domain-relevant terms. Evaluated on xBD (6,369 satellite images, 3 damage classes) and RescueNet (4,494 UAV images, 12 damage classes), VLCE demonstrated significant improvements. On RescueNet, captions generated by VLCE with a Transformer decoder were preferred over QwenVL baselines in 95.33% of image pairs by InfoMetIC and 73.64% by CLIPScore, showing enhanced factual consistency and domain-appropriate vocabulary.

Key takeaway

For Machine Learning Engineers developing domain-specific image description systems, relying solely on general-purpose VLMs like QwenVL for critical tasks such as disaster assessment is insufficient. You should integrate external semantic knowledge, such as from ConceptNet or WordNet, to augment your model's vocabulary and improve factual consistency. Consider a two-stage approach, refining initial VLM outputs with a knowledge-enriched sequential model to achieve more actionable and domain-appropriate captions. This will significantly reduce hallucinations and semantic incoherence in your generated descriptions.

Key insights

Integrating external knowledge graphs into VLMs significantly enhances domain-specific image descriptions and reduces factual errors.

Principles

• Domain knowledge improves VLM output quality.
• Two-stage refinement enhances caption accuracy.
• External semantic sources reduce VLM hallucinations.

Method

A baseline VLM generates initial captions from YOLOv8 detections, then a knowledge-enriched sequential model (CNN-LSTM or Transformer) refines them using a 1,566-term augmented vocabulary from ConceptNet/WordNet.

In practice

• Augment VLM vocabularies with domain-specific terms.
• Implement two-stage caption generation workflows.
• Integrate knowledge graphs for factual consistency.

Topics

• Vision-Language Models
• Disaster Assessment
• Image Captioning
• Knowledge Graphs
• YOLOv8
• Satellite & UAV Imagery

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning — Retrieval-augmented image captioning uses hierarchical multi-modal knowledge retrieval and LLMs to generate context-rich news image descriptions.
• Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation — Decoupling perception and reasoning in MLLMs prevents shortcut learning by ensuring visual grounding.
• WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata — WikiVQABench evaluates VLMs on knowledge-intensive VQA by requiring external knowledge beyond visual cues for correct answers.
• From Pixels to Prompts: Vision-Language Models — Understanding Vision-Language Models requires a clear mental map beyond just knowing buzzwords.
• DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments — Multi-stage multimodal reasoning is crucial for effective UAV-based disaster response, requiring specialized benchmarks and lightweight models.
• Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA — Chain-of-Procedure (CoP) enhances VLM performance in visual procedural QA by addressing retrieval and granularity issues.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.