Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

2026-07-12 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

Researchers have introduced Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR), a new set of heuristic metrics designed to quantify the abstractiveness of generated text summaries. These metrics measure the divergence from extractive copying of source text, moving beyond traditional surface-level evaluations like ROUGE. The formulation employs a harmonic mean of document lengths, adjusted by a cubic non-overlap factor, ensuring dimensionally consistent and bounded output with non-linear sensitivity to the extractive-abstractive boundary. Empirical validation on 100 XSUM documents, using four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small), demonstrated their effectiveness. The metrics successfully discriminated between extractive models, showing SA values around 0.12-0.26, and abstractive models, with SA values ranging from 0.96-1.77. Furthermore, the Abstraction Ratio proved useful in identifying summaries that might require manual review for potential hallucination. The associated code and results are publicly available.

Key takeaway

For NLP Engineers evaluating summarization models, integrating Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) into your evaluation pipeline is crucial. These metrics provide a nuanced understanding of abstractiveness beyond ROUGE, helping you accurately classify model behavior (extractive vs. abstractive). Use the Abstraction Ratio to efficiently identify summaries that warrant manual inspection for potential hallucination, streamlining your quality control process and improving model reliability.

Key insights

New metrics (RA, SA, AR) quantify summary abstractiveness, distinguishing extractive from abstractive models and flagging potential hallucination.

Principles

• Abstractiveness metrics should be dimensionally consistent and bounded.
• Non-linear sensitivity is crucial for extractive-abstractive boundary.
• Beyond ROUGE, abstractiveness quantifies summary divergence from source.

Method

The metrics are formulated using the harmonic mean of document lengths, modulated by a cubic non-overlap factor, to measure divergence from source text.

In practice

• Use SA to classify summarization models as extractive or abstractive.
• Employ AR to flag summaries for manual hallucination review.
• Integrate RA/SA/AR for deeper summarization model evaluation.

Topics

• Text Summarization
• Abstractiveness Metrics
• ROUGE Evaluation
• Natural Language Processing
• Hallucination Detection
• Model Evaluation

Code references

• katweNLP/AbstractionStudy

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• MASF: A Multi-Model Adaptive Selection Framework for Abstractive Text summarization — Combining multiple fine-tuned transformer models with an adaptive selection mechanism significantly improves abstractive summarization quality.
• AugAbEx: Bridging Abstractive and Extractive Legal Summarization — A pipeline converts abstractive legal summaries into expert-aligned extractive gold standards, overcoming manual annotation costs.
• ScholarSum: Student-Teacher Abstractive Summarization via Knowledge Graph Reasoning and Reflective Refinement — ScholarSum uses a student-teacher, graph-based approach to achieve fluent and faithful scientific abstractive summarization.
• From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models — Contextual models and unbiased evaluation significantly improve multilingual grievance detection over word-level lexicons.
• Adapting AutoARGUE for Automatic Report Evaluation under Missing Citation Annotations — Robust RAG evaluation requires specific adaptations to handle missing citation annotations effectively.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.