RETUYT-INCO at BEA 2026 Shared Task 1: Feature-Enriched mDeBERTa for Word Difficulty Prediction

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

RETUYT-INCO participated in the BEA 2026 Shared Task 1, focusing on Vocabulary Difficulty Prediction for English Learners. This regression task involved predicting GLMM psychometric difficulty scores for English words, given L1 cues in Spanish, German, or Mandarin. They submitted two systems to the closed track: a feature-engineered XGBoost regressor for all L1s, and a 3-seed mdeberta-v3-base ensemble for Spanish, enhanced with handcrafted features. Their best result was 1.094 RMSE on Spanish, a 13.0% reduction over the XLM-RoBERTa-base baseline. A LaBSE cross-lingual cosine feature significantly reduced development RMSE by 0.091. Notably, the feature-only XGBoost, without neural fine-tuning or GPU, outperformed the XLM-RoBERTa-base development baseline with 1.273 RMSE versus 1.287 RMSE.

Key takeaway

For Machine Learning Engineers developing cross-lingual NLP systems, you should prioritize robust feature engineering, especially incorporating cross-lingual embeddings like LaBSE. Your team can achieve competitive performance with simpler models like XGBoost, even without GPUs, potentially outperforming larger neural baselines. Consider prepending handcrafted features directly into transformer inputs to boost fine-tuning effectiveness.

Key insights

Feature engineering and cross-lingual embeddings significantly enhance word difficulty prediction, even outperforming complex neural baselines.

Principles

• Handcrafted features boost neural model performance.
• Cross-lingual embeddings are powerful for L1-L2 similarity.
• Simpler models can be highly competitive.

Method

The approach involved fine-tuning mDeBERTa-v3-base with prepended handcrafted features and utilizing a feature-engineered XGBoost regressor for word difficulty prediction.

In practice

• Integrate LaBSE cross-lingual cosine for L1-L2 tasks.
• Evaluate XGBoost for strong, efficient baselines.
• Prepend features as input tokens for mDeBERTa fine-tuning.

Topics

• Vocabulary Difficulty Prediction
• mDeBERTa-v3-base
• XGBoost
• Feature Engineering
• Cross-lingual Embeddings
• BEA Shared Task

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction — Soft-target loss fine-tuning significantly improves LLM and MLM performance in continuous value prediction tasks.
• Granite Embedding Multilingual R2: Open Apache 2.0 Multilingual Embeddings with 32K Context — Best Sub-100M Retrieval Quality — New IBM Granite multilingual embedding models offer superior retrieval quality and extended context for diverse language and code applications.
• NLP-based Page Classification for Efficient LLM Extraction from Brazilian Public Tender Documents — Combining NLP page classification with LLM extraction significantly reduces token count and cost for document processing.
• Machine learning and digital pragmatics: Which word category influences emoji use most? — MARBERT model shows promise in predicting Arabic emoji use, but needs further refinement for dialectal nuances.
• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts — Careful feature selection yields interpretable, generalizable models for language proficiency assessment with high accuracy.
• What Happens When a GPT Reads Your Message — Embeddings transform language into a geometric space where numerical proximity and direction encode semantic meaning and relationships.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.