Transformer-based readability classifiers are worse than you think: Evidence from cross-domain Arabic readability assessment

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A study on Arabic readability assessment introduced the Jordanian School Textbook Corpus (JSTC), comprising 82,512 segments from 240 textbooks across grades 1-12. Researchers trained XGBoost classifiers using 127 handcrafted linguistic features, fine-tuned CAMeLBERT transformers, and hybrid architectures. These models were evaluated both in-domain (JSTC combined with DARES) and on the out-of-domain BAREC benchmark. CAMeLBERT achieved strong in-domain performance with a Quadratic Weighted Kappa (QWK) of 0.830, but its cross-domain QWK significantly collapsed to 0.085. In contrast, XGBoost, relying solely on linguistic features, maintained the highest cross-domain QWK of 0.240. Interestingly, adding [CLS] embeddings to these linguistic features actively harmed transfer performance. Probing revealed CAMeLBERT layers capture some linguistic features, but higher-level signals dominate. The findings emphasize prioritizing linguistically-grounded features for robust cross-domain readability assessment.

Key takeaway

For NLP Engineers developing Arabic readability classifiers, especially for cross-domain applications, you should prioritize incorporating linguistically-grounded features. Relying solely on fine-tuned Transformer models like CAMeLBERT risks significant performance collapse when deployed on unseen text domains. Instead, consider hybrid architectures or models like XGBoost that leverage explicit linguistic features, which demonstrated superior cross-domain robustness (QWK 0.240 vs. 0.085). This approach ensures your models generalize effectively beyond their training data.

Key insights

For robust cross-domain Arabic readability assessment, linguistically-grounded features are superior to contextual embeddings.

Principles

• Cross-domain robustness demands linguistically-grounded features.
• Contextual embeddings may collapse out-of-domain.
• Adding [CLS] embeddings can harm transfer.

Method

Researchers trained XGBoost classifiers using 127 handcrafted linguistic features and fine-tuned CAMeLBERT transformers. Evaluation occurred both in-domain (JSTC + DARES) and on the BAREC out-of-domain benchmark.

In practice

• Prioritize linguistic features for Arabic readability.
• Evaluate models on out-of-domain benchmarks.
• Analyze token importance like nouns and "al-".

Topics

• Arabic NLP
• Readability Assessment
• Transformers
• XGBoost
• Linguistic Features
• Cross-domain Evaluation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• An End-to-End Hybrid Framework for Rumour Detection in Low-Resources Algerian Dialect — A hybrid framework combining transformer embeddings with classical classifiers enables effective rumour detection in low-resource Algerian dialect.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• Geometry of Semantic Space: Comparative Study of Discrete and Continuous Models — Graph-based semantic models offer more interpretable meaning organization than transformer embeddings, despite similar local topology.
• State-of-the-Art Transformer Pipelines in spaCy — spaCy offers robust, configurable Transformer pipelines for NLP tasks, often outperforming LLMs on predictive tasks with modest data.
• Scripts Through Time: A Survey of the Evolving Role of Transliteration in NLP — Transliteration bridges the "script barrier" in NLP, enhancing cross-lingual transfer and efficiency, especially for low-resource languages.
• NLP-based Page Classification for Efficient LLM Extraction from Brazilian Public Tender Documents — Combining NLP page classification with LLM extraction significantly reduces token count and cost for document processing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.