Position: What Are We Measuring? Rethinking Evaluation in Natural Language Generation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Research Methodology & Innovation · Depth: Expert, quick

Summary

The paper "Position: What Are We Measuring? Rethinking Evaluation in Natural Language Generation" by Wajdi Zaghouani, presented at the Fifth Workshop on Generation, Evaluation and Metrics (GEM) in July 2026, argues that current Natural Language Generation (NLG) evaluation metrics lack a coherent theory of measurement. It identifies a fundamental construct validity problem, where metrics are treated as proxies for output quality without explicitly defining the underlying constructs they operationalize. The analysis examines four dominant evaluation paradigms: reference-based metrics, embedding-based metrics, LLM-as-judge, and human evaluation, demonstrating how each conflates construct definition with operationalization. Drawing on psychometric traditions from Cronbach and Meehl (1955) and recent NLP applications, the paper proposes adopting a measurement modeling perspective for NLG evaluation, foundational concepts like construct validity, reliability, and consequential validity, and outlines a preliminary taxonomy of NLG quality constructs.

Key takeaway

For NLP Engineers and Research Scientists designing or selecting NLG evaluation metrics, you should critically assess whether your chosen metrics explicitly define the underlying quality constructs they aim to measure. Implement a measurement modeling perspective, focusing on construct, reliability, and consequential validity to ensure your evaluations accurately reflect desired output quality. This approach will lead to more robust and interpretable NLG system comparisons and development.

Key insights

NLG evaluation metrics suffer from a construct validity problem, conflating definition with operationalization, requiring a measurement modeling perspective.

Principles

• Metrics must explicitly define constructs.
• Adopt measurement modeling perspective.
• Ensure construct, reliability, consequential validity.

Method

Proposes adopting a measurement modeling perspective for NLG evaluation, borrowing concepts of construct validity, reliability, and consequential validity, and outlining a preliminary taxonomy of NLG quality constructs.

In practice

• Apply psychometric traditions to NLG.
• Develop explicit construct definitions.
• Re-evaluate existing metrics for validity.

Topics

• Natural Language Generation
• Evaluation Metrics
• Construct Validity
• Measurement Theory
• Psychometrics
• LLM-as-Judge

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics — Enriched Meaning Representations improve NLG quality, especially for complex tasks and small, variable datasets.
• Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025 — NLP human annotation reporting often lacks critical validity details, hindering reliability.
• Questions from readers of my book — Data quality is paramount for Neural NLG success, often outweighing model architecture.
• GENIE: A Fine-Grained Measure for Novelty — GENIE offers a fine-grained metric to measure task-specific novelty in LLM outputs, surpassing holistic evaluations.
• Evaluation Revisited: A Taxonomy of Evaluation Concerns in Natural Language Processing — A new taxonomy synthesizes historical and contemporary NLP evaluation concerns across four key dimensions.
• Synthetic but Not Realistic: The Evaluation Challenge in Generative Modelling for Structured Electronic Medical Records — Evaluating synthetic EMR data requires a multi-dimensional framework beyond statistical similarity to ensure clinical validity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.