ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

2026-04-12 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

ALBA is a new, linguistically grounded benchmark introduced to evaluate Large Language Model (LLM) performance specifically in European Portuguese (pt-PT). Developed by Inês Vieira et al. for PROPOR 2026, ALBA addresses the current imbalance where most existing training data and benchmarks for Portuguese are in Brazilian Portuguese (pt-BR). The benchmark assesses LLM proficiency across eight distinct linguistic dimensions: Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. Constructed manually by language experts, ALBA integrates an "LLM-as-a-judge" framework to enable scalable evaluation of pt-PT generated language. Initial experiments using a diverse set of models revealed significant performance variability across these linguistic dimensions, underscoring the critical need for more comprehensive, variety-sensitive benchmarks to advance pt-PT language tools.

Key takeaway

For research scientists developing or deploying LLMs for multilingual applications, you should prioritize using variety-specific benchmarks like ALBA for European Portuguese. This ensures your models accurately reflect the nuances of target language varieties, avoiding performance degradation from over-reliance on dominant dialects. Integrate such benchmarks into your evaluation pipelines to identify and address linguistic shortcomings, fostering more robust and culturally appropriate LLM development.

Key insights

ALBA provides a linguistically-grounded benchmark for European Portuguese LLM evaluation, addressing a critical language variety gap.

Principles

• Variety-specific benchmarks are crucial for under-represented languages.
• Expert-crafted data improves linguistic evaluation accuracy.

Method

ALBA is manually constructed by language experts and uses an "LLM-as-a-judge" framework for scalable evaluation of European Portuguese LLM outputs across eight linguistic dimensions.

In practice

• Evaluate LLMs on specific language varieties.
• Use expert-curated datasets for nuanced linguistic assessment.

Topics

• European Portuguese (pt-PT)
• Large Language Models
• Linguistic Benchmarking
• LLM-as-a-Judge Framework
• Language Variety Evaluation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs — Most LLMs exhibit a strong bias towards Brazilian Portuguese, highlighting the need for balanced multilingual representation.
• A Larger Annotated Corpus of Portuguese Coreference — LLMs can annotate Portuguese coreference, but a new LLM-based system did not outperform existing rule-based methods.
• MARCA: A Checklist-Based Benchmark for Multilingual Web Search — MARCA evaluates LLM web search reliability using bilingual questions and checklist rubrics.
• PorTEXTO: A European Portuguese Benchmark for Visual Text Extraction — Specialized multilingual data significantly improves European Portuguese OCR performance over model size or resolution.
• Causal_QA.PT: A Human–LLM Co-Curated Benchmark for Causal Question Answering in Portuguese Language — A new human-LLM co-curated benchmark improves causal reasoning evaluation for Portuguese LLMs.
• MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation — MORPHOGEN evaluates LLM gender-aware morphological generation across French, Arabic, and Hindi, revealing performance gaps and biases.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.