From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, extended

Summary

A new study addresses the limitations of word-level lexicons for multilingual grievance detection in online text, which often fail to account for context like negation or quotation. Researchers found that existing evaluation pools for the Grievance Dictionary were circular, inflating its macro-AUROC of 0.686 to a 0.500 floor. To counter this, they developed context-reading models, maintaining the dictionary's 22-construct ontology, and evaluated them on a non-circular, five-language benchmark. The full-post mDeBERTa model significantly improved average precision, particularly on lexicon-negative text, from 0.142 to 0.198 (a 39% relative gain), with the largest improvements on quoted, implicit, and cross-sentence grievances. The study also established a 12.9% unconditional prevalence of grievance constructs. Code and benchmark are publicly available at https://github.com/behavioral-ds/multilingual_grievance.

Key takeaway

For NLP Engineers or Threat Assessment Analysts building grievance detection systems, relying on word-level lexicons or evaluation on lexicon-selected data is insufficient. Your models will miss critical context-dependent signals like quoted or implicit grievances, leading to poor recall. You should integrate full-post contextual models and validate them against non-circular, unconditionally sampled benchmarks to ensure accurate and comprehensive detection. This approach provides a more trustworthy assessment of model performance and real-world prevalence.

Key insights

Contextual models measure grievance more faithfully than word-level lexicons, especially on non-lexicon-selected text.

Principles

• Lexicon-conditioned evaluation distorts true model performance.
• Grievance is a latent, context-dependent construct.
• Multilingual construct validity requires per-language evaluation.

Method

Replace lexical term matching with contextual models (e.g., mDeBERTa-v3-base) that read the full post, not just the target sentence, to predict 22 grievance constructs. Evaluate on a non-circular, stratified benchmark.

In practice

• Use full-post encoders for nuanced grievance detection.
• Test models on unconditionally sampled text for true prevalence.
• Consider context for quoted, implicit, or cross-sentence expressions.

Topics

• Grievance Detection
• Contextual Models
• Multilingual NLP
• Lexical Analysis
• Evaluation Benchmarks
• Threat Assessment

Code references

• behavioral-ds/multilingual_grievance

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models — Contextual models and unbiased evaluation significantly improve multilingual grievance detection over word-level lexicons.
• Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale — Human subjectivity annotation is inherently variable and influenced by context, while LLMs offer plausible but distinct scoring.
• Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages — Idiomatic expressions challenge multilingual NLP, with literal meanings and low-resource languages posing significant hurdles.
• Revisiting Faithfulness Annotations for Long-form Summaries — Existing long-form summary faithfulness benchmarks contain significant discourse-level annotation errors impacting system evaluations.
• The BD-LSC Dataset: Facilitating the Benchmarking of Models for Lexical Semantic Change Detection in Slang and Standard Usage — New benchmarks address bi-directional lexical semantic change and slang usage, revealing challenges with rare slang senses.
• Multi-Stage Training for Abusive Comment Detection in Indic Languages — Multi-stage training with language preprocessing and model ensembles improves abusive comment detection while minimizing false positives.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.