Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News

2026-04-12 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A study presented at PROPOR 2026 evaluates reference-free summarization quality metrics for Portuguese financial news, addressing the lack of reliable evaluation methods in this specialized domain. Researchers João Victor Assaoka Ribeiro, Thomas Pires Correia, José Vitor Souza Cardoso Requena, and Lilian Berton compared Question Answering (QA) based metrics against a direct LLM-as-a-Judge baseline. Their pipeline incorporated Lexical, Binary, and Semantic (LLM-based) QA scoring methods, validated against a human ground truth of 50 news items annotated for Faithfulness and Completeness. The findings indicate that granular QA metrics significantly outperform the monolithic LLM-Judge for evaluating Completeness, with QA-Binary achieving a rank correlation of \u03c1 \u2248 0.49. For Faithfulness, the Semantic QA metric demonstrated a "super-human" ability to detect subtle hallucinations, such as temporal shifts, that human annotators missed.

Key takeaway

For research scientists developing or evaluating automatic summarization systems for specialized languages like Portuguese, you should prioritize granular Question Answering (QA) based metrics over monolithic LLM-as-a-Judge approaches. Specifically, integrate QA-Binary for robust completeness assessment and Semantic QA for detecting subtle faithfulness issues, as these methods offer superior correlation with human perception and even "super-human" detection capabilities for hallucinations in financial news.

Key insights

Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.

Principles

• Granular QA metrics improve completeness evaluation.
• Semantic QA detects subtle hallucinations.
• Human evaluation can have ceiling effects.

Method

The study proposes a pipeline comparing Lexical, Binary, and Semantic QA scoring methods against an LLM-as-a-Judge baseline, validated with human judgments on 50 Portuguese financial news summaries for faithfulness and completeness.

In practice

• Use QA-Binary for completeness evaluation.
• Employ Semantic QA to detect subtle hallucinations.
• Consider atomic QA pairs for specialized domains.

Topics

• Reference-Free Summarization
• Summarization Evaluation Metrics
• LLM-as-a-Judge
• Question Answering Metrics
• Portuguese Financial News

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• The Inadequacy of Automatic Evaluation Metrics in Question Answering: A Case-Study in Portuguese — Traditional lexical metrics are insufficient for evaluating Question Answering quality, especially in non-English languages.
• Summarisation and Knowledge Distillation — How Agents Summarise Without Hallucinating — Robust LLM summarization requires architectural patterns to prevent hallucinations and ensure factual grounding and completeness.
• Structured Summaries for Retrieval-Augmented Generation in Portuguese-Language Consumer Complaints — Structured summaries significantly improve dense retrieval effectiveness in RAG systems for noisy, lengthy texts.
• Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding — ConSUM enhances summary factuality by combining source consistency with candidate consensus via MBR reranking.
• From 50K to 8.2 Million in 24 Hours: Vozinha's Algorithmic Consecration and the Multilingual Making of World Cup Visibility — Algorithmic visibility for public figures is shaped by distinct cross-lingual narrative frames and platform metrics.
• Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants — LLMs exhibit significant performance gaps in culturally-grounded, dialectal Arabic, especially with open-ended questions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.