MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection

2026-06-10 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study addresses the challenges of detecting mental health disorders from Arabic social media text, including dialectal variation, informal language, and limited annotated resources. It proposes a two-phase framework for Arabic mental health text classification. Phase 1 involves Domain-Adaptive and Task-Adaptive Pretraining (DAPT and TAPT) of AraBERT, CAMeLBERT, and MARBERT using a large corpus of unlabeled Arabic mental health tweets to identify the most effective backbone model. Phase 2 evaluates the selected model across four configurations, combining single-stage and hierarchical two-stage classification architectures with full fine-tuning and Low-Rank Adaptation (LoRA). The researchers constructed a novel annotated Arabic mental health dataset of 50,670 tweets across six categories, achieving strong inter-annotator agreement (Krippendorff's Alpha = 0.733). Experimental results show that the domain-adapted MARBERT, named MentalMARBERT, significantly improves performance. The hierarchical two-stage architecture with full fine-tuning achieved the best overall performance, with a macro-F1 of 0.861 and an accuracy of 0.877.

Key takeaway

For NLP Engineers developing Arabic mental health detection systems, this research indicates that domain-adaptive pre-training and hierarchical classification are critical for achieving high performance. You should prioritize fine-tuning models like MARBERT on domain-specific data and implement a two-stage classification architecture. This approach, demonstrated by MentalMARBERT's 0.861 macro-F1, offers a robust strategy to overcome dialectal challenges and class imbalance in Arabic social media text.

Key insights

Domain-adaptive pre-training and hierarchical classification significantly improve Arabic mental health disorder detection.

Principles

• Domain-adaptive pre-training enhances model performance.
• Hierarchical classification improves multi-class detection.
• High-quality annotated datasets are crucial.

Method

A two-phase framework: DAPT/TAPT on pre-trained models, then evaluate selected model with hierarchical two-stage classification and full fine-tuning.

In practice

• Apply DAPT/TAPT for domain-specific NLP tasks.
• Consider hierarchical architectures for multi-class problems.
• Develop high-quality, domain-specific datasets.

Topics

• Mental Health Detection
• Arabic NLP
• Domain-Adaptive Pre-training
• Hierarchical Classification
• MARBERT
• Text Classification

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• An End-to-End Hybrid Framework for Rumour Detection in Low-Resources Algerian Dialect — A hybrid framework combining transformer embeddings with classical classifiers enables effective rumour detection in low-resource Algerian dialect.
• LinguIUTics at PsyDefDetect: Iterative Imbalance-Aware Fine-tuning of Qwen3-8B for Psychological Defense Mechanism Classification — Iterative imbalance-aware fine-tuning of LLMs, combined with augmentation and post-processing, significantly improves rare-class NLP performance.
• Healthsea: an end-to-end spaCy pipeline for exploring health supplement effects — Healthsea is a spaCy pipeline that uses ML/NLP to extract health effects from supplement user reviews.
• Multi-Stage Training for Abusive Comment Detection in Indic Languages — Multi-stage training with language preprocessing and model ensembles improves abusive comment detection while minimizing false positives.
• KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts — Domain-specific pre-training significantly enhances BERT models for Norwegian clinical NLP tasks.
• Multi-View Attention Multiple-Instance Learning Enhanced by LLM Reasoning for Cognitive Distortion Detection — Combining LLMs with MIL and ELB components improves cognitive distortion detection by enhancing interpretability and reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.