_alexcristea@EEUCA 2026: A Robust Early-Fusion ERNIE Pipeline for Multimodal COVID-19 Vaccine Meme Classification

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Advanced, medium

Summary

A new system for the EEUCA0022026 shared task on Multimodal Vaccine Critical Meme Detection categorizes social media memes from the VaxMeme dataset into Vaccine Critical, Neutral, or Pro-Vaccine stances. This robust text-fusion pipeline addresses challenges like sarcasm and label noise by extracting text from images via OCR and combining it with social media posts. The unified text is then processed by an ERNIE 2.0-Large encoder. To mitigate severe overfitting common in subjective meme datasets, the system employs a Multi-Sample Dropout architecture, averaging predictions across five parallel dropout masks with p = 0.3. This optimized, lightweight text-only approach achieved a peak Macro F1 score of 0.834. An ablation study further demonstrated that standard Cross-Entropy loss offers superior robustness against inherent label noise compared to Focal Loss.

Key takeaway

For Machine Learning Engineers developing social media content classification systems, consider a text-centric approach for multimodal data. Your models can achieve high performance, like a 0.834 Macro F1, by using OCR to extract image text and fusing it with post content, processed by models such as ERNIE 2.0-Large. Implement Multi-Sample Dropout (p=0.3) to combat overfitting on subjective datasets and prefer standard Cross-Entropy loss for superior robustness against label noise.

Key insights

A text-only ERNIE 2.0-Large pipeline with Multi-Sample Dropout effectively classifies vaccine memes, outperforming visual encoders and Focal Loss.

Principles

• Text-based analysis can surpass visual methods for meme classification.
• Multi-Sample Dropout improves robustness against overfitting in subjective data.
• Standard Cross-Entropy offers better label noise robustness than Focal Loss.

Method

Extract text from images via OCR, concatenate with social media posts, process with ERNIE 2.0-Large, then use a Multi-Sample Dropout classification head (p=0.3) averaging five parallel masks.

In practice

• Use OCR for image text in multimodal tasks.
• Implement Multi-Sample Dropout for subjective datasets.
• Prioritize Cross-Entropy for noisy label environments.

Topics

• Multimodal Classification
• Vaccine Meme Detection
• ERNIE 2.0-Large
• Optical Character Recognition
• Multi-Sample Dropout
• Label Noise Robustness

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memes — Reinforcement learning with CoT supervision significantly improves MLLM performance and explainability for hateful meme detection.
• A Multimodal and Explainable Machine Learning Approach to Diagnosing Multi-Class Ejection Fraction from Electrocardiograms — Multimodal machine learning can accurately stratify LVEF from ECG and EHR data, enhancing accessibility.
• MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment — MIC enhances multi-scale representations by aligning isotropic subspaces and mitigating redundancy through regularization and self-distillation.
• AAbAAC: An Annotated Corpus for Autoimmunity Information Extraction — Small, specialized annotated corpora significantly boost information extraction performance in niche biomedical domains.
• Log-Likelihood, Simpson's Paradox, and the Detection of Machine-Generated Text — Inappropriate aggregation of token-level likelihood scores causes Simpson's paradox, hindering machine-generated text detection.
• GigaTIME: Scaling tumor microenvironment modeling using virtual population generated by multimodal AI — GigaTIME uses multimodal AI to convert common H&E slides into virtual mIF images, enabling population-scale precision immuno-oncology research.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.