Glite at BEA 2026 Shared Task 1: Holistic Difficulty Models Dominate, Feature Engineering Closes the Gap in L1-Aware Vocabulary Difficulty Prediction

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Glite's submission to the BEA 2026 Shared Task on L1-Aware English Vocabulary Difficulty Prediction achieved top rankings, securing 1st in the closed track and 2nd in the open track across Spanish, German, and Mandarin L1s. Their systems significantly reduced baseline RMSE by an average of 29.9% in the closed track and 35.9% in the open track. The approach utilized per-L1 CatBoost regressors, incorporating 1,161 linguistic, psycholinguistic, dictionary, and LLM-derived features from 129 sets. Feature selection employed Recursive Feature Elimination with nested cross-validation, yielding compact models with 29-150 features. For the closed track, a compliance audit deemed 57 of 129 feature sets eligible. In the open track, decoder-LLM LoRA regression heads, particularly LLaMA-3.1-8B, delivered the most substantial performance improvements. A simpler per-L1 CatBoost model on RFE-selected features proved competitive with or superior to Ridge-stacking ensembles.

Key takeaway

For Machine Learning Engineers developing L1-aware vocabulary difficulty prediction systems, prioritize extensive feature engineering and integrate LLM-derived features. Your models should leverage per-L1 CatBoost regressors with Recursive Feature Elimination for compact, high-performing solutions. Consider LLaMA-3.1-8B LoRA regression heads for substantial performance improvements, as simpler CatBoost models can outperform complex Ridge-stacking ensembles. This approach significantly reduces prediction error, as demonstrated by a 35.9% RMSE reduction.

Key insights

Holistic difficulty models, especially with LLM features and RFE, significantly improve L1-aware vocabulary difficulty prediction.

Principles

• L1-specific models enhance prediction accuracy.
• Extensive feature engineering boosts performance.
• LLM-derived features offer substantial gains.

Method

Build per-L1 CatBoost regressors using RFE-selected linguistic, psycholinguistic, dictionary, and LLM features, with out-of-fold LLM predictions as additional inputs.

In practice

• Apply RFE for compact, effective feature sets.
• Integrate LLaMA-3.1-8B LoRA heads for gains.
• Consider CatBoost over complex ensembles.

Topics

• L1-Aware Vocabulary Prediction
• CatBoost Regressors
• Recursive Feature Elimination
• Large Language Models
• LLaMA-3.1-8B
• Feature Engineering

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget — Compute-aware LLM evaluation requires analyzing full training trajectories, not just final performance metrics.
• LLMs on Tabular Data with Limited Semantics: Evidence from Industrial Car Retrofit Prediction — LLMs serve best as complementary components to strong tabular machine learning baselines, not replacements, for industrial data with limited semantics.
• A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget — Compute-constrained LLM training trajectories reveal non-monotonic degradation and instability, challenging endpoint-only evaluation.
• What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction — Soft-target loss fine-tuning significantly improves LLM and MLM performance in continuous value prediction tasks.
• Quantization-Robust LLM Unlearning via Low-Rank Adaptation — Low-rank adaptation (LoRA) enables effective LLM unlearning that survives aggressive post-training quantization.
• Scikit-LLM vs. Traditional Text Classifiers: When Should You Use an LLM? — LLMs offer superior zero-shot text classification accuracy and competitive latency, especially with limited data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.