Multimodal Transformer Framework for Multilingual Harmful Meme Classification

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new multimodal transformer-based framework is introduced for multilingual harmful meme classification. This framework addresses the challenge of detecting misogynistic, homophobic, and transphobic content across diverse languages and cultures on social media. The architecture integrates XLM-RoBERTa for multilingual text encoding and the Swin Transformer for hierarchical visual feature extraction. A cross-attention fusion mechanism facilitates interaction between these textual and visual modalities. The combined representation is then fed into a classification layer for multi-class prediction. Experiments were conducted across multiple datasets, covering eight languages and three harmful content categories: misogyny, homophobia/transphobia, and hate speech. These experiments demonstrate consistent improvements over baseline multimodal systems. Evaluated using the macro-F1 score, the framework proves effective in both high-resource and low-resource languages, supporting robust multilingual systems for online harmful content detection.

Key takeaway

For AI Scientists or Machine Learning Engineers developing content moderation systems, this multimodal transformer framework offers a robust approach for multilingual harmful meme detection. You should consider integrating XLM-RoBERTa for text, Swin Transformer for visual features, and a cross-attention mechanism for fusion. This architecture consistently improves detection performance across eight languages and various harmful categories, including low-resource contexts. Implementing this can significantly enhance the accuracy and inclusivity of your moderation efforts.

Key insights

A multimodal transformer framework using XLM-RoBERTa and Swin Transformer with cross-attention significantly improves multilingual harmful meme classification.

Principles

• Multimodal fusion is crucial for meme detection.
• Cross-attention enhances modality interaction.
• Transformers effectively capture harmful signals.

Method

XLM-RoBERTa encodes multilingual text, Swin Transformer extracts hierarchical visual features. A cross-attention mechanism fuses these, followed by a classification layer for multi-class prediction.

In practice

• Employ XLM-RoBERTa for multilingual text.
• Use Swin Transformer for visual features.
• Integrate cross-attention for modality fusion.

Topics

• Multimodal Transformers
• Harmful Content Detection
• Meme Classification
• XLM-RoBERTa
• Swin Transformer
• Cross-attention
• Multilingual NLP

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection — KITE integrates text, images, and knowledge graphs via a tri-modal transformer for enhanced fake news detection and interpretability.
• From TF-IDF to Transformers: A Comparative and Ensemble Approach to Sentiment Classification — A soft voting ensemble of diverse machine learning and transformer models significantly enhances sentiment classification accuracy, particularly for nuanced text.
• mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection — Finetuning mid-sized LLMs with QLoRA and augmented data improves multilingual online polarization detection.
• Unified Safe In-context Image Generation in Multimodal Diffusion Transformers via Restricting Unsafe Information Flows — UVR restricts unsafe information flow in DiT multimodal attention during an early "start-up stage" to prevent harmful image generation.
• MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models — MLLM-Microscope reveals MLLM internal representation dynamics, showing modality fusion impacts linearity, dimension, and anisotropy across transformer layers.
• HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models — Harmful video understanding in LVLMs requires multi-layered evaluation beyond binary classification and explicit rationales.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.