Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding

2026-05-28 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

ConSUM is a proposed system designed to improve the factuality of model-generated summaries by addressing limitations in existing reranking methods. Unlike approaches that solely use the source document for guidance, ConSUM reranks candidate summaries based on two key factors: consistency with the source document and consensus among other generated candidates. It establishes consensus using Minimum Bayes Risk (MBR) decoding across the set of generated summaries, while ensuring consistency through factuality-aware metrics that compare the summary against its source. Rigorous testing indicates ConSUM is competitive with current methods, and human evaluations further confirm its summaries are preferred. The system's code is publicly available at https://github.com/naist-nlp/ConSUM.

Key takeaway

For NLP Engineers focused on improving summary factuality, ConSUM offers a robust reranking strategy. You should consider integrating Minimum Bayes Risk (MBR) decoding with factuality-aware consistency checks into your summarization pipelines. This approach, validated by human evaluations, can significantly enhance the reliability of your model-generated outputs, moving beyond source-only guidance. Explore the provided code to adapt this method for your specific applications.

Key insights

ConSUM enhances summary factuality by combining source consistency with candidate consensus via MBR reranking.

Principles

• Factuality in summarization benefits from multi-candidate evaluation.
• Reranking can integrate source consistency and candidate consensus.
• MBR decoding is effective for establishing summary consensus.

Method

ConSUM reranks candidate summaries by applying Minimum Bayes Risk (MBR) decoding for consensus and factuality-aware metrics for source consistency.

In practice

• Implement MBR decoding for summary candidate selection.
• Use factuality-aware metrics to check source consistency.
• Explore multi-candidate reranking for improved summary quality.

Topics

• Text Summarization
• Factuality
• Minimum Bayes Risk
• Reranking
• Natural Language Processing
• ConSUM

Code references

• naist-nlp/ConSUM

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• DWTSumm: Discrete Wavelet Transform for Document Summarization — DWT-based semantic compression improves LLM summarization by preserving factual grounding and reducing hallucinations in long, domain-specific texts.
• Consistency Analysis of Sentiment Predictions using Syntactic & Semantic Context Assessment Summarization (SSAS) — SSAS improves LLM sentiment prediction consistency by pre-processing noisy data into structured, high-signal prompts.
• BiCon-Gate: Consistency-Gated De-colloquialisation for Dialogue Fact-Checking — BiCon-Gate improves dialogue fact-checking by consistently de-colloquializing claims with contextual semantic gating.
• Summarisation and Knowledge Distillation — How Agents Summarise Without Hallucinating — Robust LLM summarization requires architectural patterns to prevent hallucinations and ensure factual grounding and completeness.
• RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora — Existing RAG benchmarks misrepresent real-world performance due to high document redundancy and similarity in enterprise corpora.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.