Validity of LLMs as data annotators: AMALIA on authority

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Portugal's AMALIA, a publicly funded 9B-parameter language model for European Portuguese, demonstrates competitive agreement with human coders on tasks like identifying the moral foundation of authority, performing within six F1 points of open models eight to thirteen times its size. However, this agreement signifies reliability, not theoretical validity, especially for constructs requiring inference beyond surface features. Researchers tested this using a "recovery gap" method, decomposing holistic prompts into atomic clauses and recombining them by theoretical rules. The study found that for one construct and corpus, AMALIA-9B's holistic performance was only about half recovered by decomposition, indicating reliance on surface correlates like moral outrage. An open multilingual LLM closed this gap on the same Portuguese corpus, suggesting the issue lies with the construct-model instrument rather than the corpus. While AMALIA can screen and pre-code at scale, it cannot yet accurately measure this specific construct alone, highlighting the need for sovereign-LLM benchmarks to test evidential routes of agreement, not just agreement itself.

Key takeaway

For NLP Engineers evaluating national language models like AMALIA for data annotation, you should prioritize validity testing beyond simple agreement metrics. If your application requires inferring complex theoretical constructs, ensure the model follows the construct's underlying theory, not just surface correlates. Implement tests like the "recovery gap" to verify the evidential route of agreement, preventing reliance on models that only pre-code rather than accurately measure.

Key insights

LLM agreement with humans doesn't guarantee theoretical validity; test the evidential route.

Principles

• Agreement (reliability) differs from validity for inferred constructs.
• Decomposing prompts tests if LLMs follow construct theory.
• Sovereign LLM benchmarks need validity tests, not just agreement.

Method

Test LLM validity by measuring the "recovery gap": compare holistic prompt performance to decomposed prompt performance, recombined by theoretical rules.

In practice

• Use AMALIA for screening and pre-coding tasks.
• Scrutinize LLM annotations for surface correlate reliance.
• Employ recovery gap testing for construct validity.

Topics

• LLM Validity
• Data Annotation
• AMALIA
• European Portuguese
• Moral Foundations Theory
• Recovery Gap
• Sovereign LLMs

Best for: Research Scientist, AI Scientist, NLP Engineer, Data Scientist

Related on AIssential

• Validity of LLMs as data annotators: AMALIA on authority — LLM agreement with human annotators does not guarantee construct validity; inferential path matters.
• AMALIA: A Fully Open Large Language Model for European Portuguese — Targeted training and native benchmarking are crucial for underrepresented language variants like European Portuguese.
• LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation — LLMs can generate synthetic data to validate NLP evaluation metrics, replacing costly human annotations.
• Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms — LLM factual recall is sensitive to entity surface forms, highlighting the need for diverse evaluation methods.
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — LLM-as-a-Judge reliability is often overstated due to reliance on flawed metrics and unaddressed biases.
• When LLMs Disagree with Human Experts: Understanding LLM Annotation Failures in Nutrition Misinformation through Hierarchical Error Analysis using Seed Oil Narratives — Large language models frequently misinterpret nuanced health misinformation, leading to annotation failures compared to human experts.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.