Validity of LLMs as data annotators: AMALIA on authority

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Social Sciences & Behavioral Studies · Depth: Expert, extended

Summary

Portugal's publicly funded AMALIA, a 9-billion-parameter language model for European Portuguese released in July 2026, demonstrates high agreement with human coders on the "authority" moral construct, performing within six points of models eight to thirteen times its size. However, a study using 748 texts (448 for confirmatory tests) reveals a significant "recovery gap" in its construct validity. Under Portuguese instructions, AMALIA's recovery gap (ΔApt) was +0.358, and +0.436 with English instructions, indicating it often relies on surface correlates rather than the theory-defined inferential path. Error analysis showed 78% of false positives were due to surface correlates, and 54% of cited evidence was partially grounded. The detection clause fired on only 17.9% (pt-PT) and 12.3% (en) of texts. This suggests AMALIA can reliably annotate but struggles to measure theoretical constructs according to their underlying theory, posing challenges for sovereign LLM programs.

Key takeaway

For AI Scientists and Policy Makers evaluating national LLM programs, you must prioritize construct validity over mere agreement when deploying models for theoretical construct measurement. Your LLM might reliably produce correct codes but for the wrong reasons, leading to invisible failures. Implement grain calibration and recovery gap analysis to audit whether your models measure constructs according to their underlying theory, ensuring robust and auditable AI instruments for critical applications.

Key insights

LLM agreement with human annotators does not guarantee construct validity; inferential path matters.

Principles

• LLM reliability (agreement) differs from validity (faithfulness to theory).
• Construct validity requires empirical evidence of theory-defined measurement.
• Recovery gap quantifies divergence between holistic and decomposed performance.

Method

Grain calibration systematically probes LLM inference by decomposing prompts into atomic clauses, consolidating answers, and quantifying the "recovery gap" to assess construct validity.

In practice

• Use grain calibration to validate LLMs for theoretical construct annotation.
• Decompose complex constructs into binary clauses for LLM evaluation.
• Audit sovereign LLMs for construct validity, not just agreement.

Topics

• LLM Validity
• Data Annotation
• Construct Validity
• Sovereign LLMs
• European Portuguese
• Grain Calibration

Code references

• sejkko/amalia-llm-on-authority

Best for: AI Scientist, Research Scientist, Policy Maker

Related on AIssential

• Validity of LLMs as data annotators: AMALIA on authority — LLM agreement with humans doesn't guarantee theoretical validity; test the evidential route.
• When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals — Agreement among LLMs or within a model's samples is an unreliable, conditional proxy for correctness, not a standalone confidence signal.
• When LLMs Disagree with Human Experts: Understanding LLM Annotation Failures in Nutrition Misinformation through Hierarchical Error Analysis using Seed Oil Narratives — Large language models frequently misinterpret nuanced health misinformation, leading to annotation failures compared to human experts.
• Dimension-Level Intent Fidelity Evaluation for Large Language Models: Evidence from Structured Prompt Ablation — Dimension-level intent fidelity evaluation uncovers LLM quality deficits missed by holistic assessment.
• LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation — LLMs can generate synthetic data to validate NLP evaluation metrics, replacing costly human annotations.
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — LLM-as-a-Judge reliability is often overstated due to reliance on flawed metrics and unaddressed biases.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.