Susmitha@LT-EDI 2026: Detecting LGBTQ+ Phobia in Multilingual Memes via Joint Representation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Computer Vision · Depth: Advanced, medium

Summary

A multimodal architecture was developed and evaluated at LT-EDI 2026 for detecting LGBTQ+ phobia in multilingual social media memes. The system uses a late-fusion strategy, where XLM-RoBERTa encodes OCR-extracted text into a representation h_t ∈ ℝ⁷⁶⁸ and CLIP extracts visual features h_v ∈ ℝ⁵¹². These are concatenated into a joint vector z = [h_t ⊕ h_v] ∈ ℝ¹²⁸⁰ and processed by a non-linear multilayer perceptron. The system achieved 3rd rank in the Chinese track (Macro F1: 0.7371) and 4th rank in the English track (Macro F1: 0.6121). However, performance in the Hindi track was significantly lower (Macro F1: 0.1616), attributed to script complexity and class imbalance. This highlights the effectiveness of global transformer models in high-resource contexts but also the need for specialized linguistic refinement in low-resource environments.

Key takeaway

For machine learning engineers developing hate speech detection systems, this research indicates that while global transformer-based models like XLM-RoBERTa and CLIP are effective for high-resource languages, you must prioritize specialized linguistic refinement and data balancing for low-resource or complex script environments. Your deployment strategy should account for significant performance drops in languages like Hindi without targeted optimization.

Key insights

Multimodal detection of LGBTQ+ phobia in memes is effective with global transformers but struggles with low-resource languages.

Principles

• Late-fusion multimodal architectures combine text and visual features effectively.
• Global transformer models excel in high-resource language contexts.

Method

OCR-extracted text is encoded by XLM-RoBERTa, visual features by CLIP. These are concatenated into a joint vector and processed via a non-linear multilayer perceptron for classification.

In practice

• Use XLM-RoBERTa for multilingual text encoding.
• Employ CLIP for robust visual feature extraction.

Topics

• Multimodal AI
• Hate Speech Detection
• Meme Analysis
• XLM-RoBERTa
• CLIP
• Cross-lingual NLP
• Transformer Models

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages — Multimodal LLMs must integrate text, speech, and vision for low-resource languages, moving beyond English-centric benchmarks.
• TeamHerald@CHIPSAL 2026: Hate Speech Detection and Sentiment Analysis of Nepali Memes using Transformer-based Architectures and Ensemble Learning — Ensemble strategies for meme analysis perform differently for binary vs. multi-class tasks, requiring task-specific aggregation.
• Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task — Supervised models with multilingual embeddings and gradient boosted trees excel at hate speech detection.
• LLM-based Detection of Manipulative Political Narratives — LLM-driven pipelines can effectively identify and cluster manipulative political narratives in raw social media data.
• mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection — Finetuning mid-sized LLMs with QLoRA and augmented data improves multilingual online polarization detection.
• Robust Dual-Signal Fusion: Hybrid Neuro-Symbolic Gating with Compressed Chain-of-Thought Refinement for Irony Detection in Social Media Texts — Hybrid neuro-symbolic fusion with compressed Chain-of-Thought effectively overcomes LLM literalism for zero-shot irony detection.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.