Frame In, Frame Out: Measuring Framing Bias in LLM-Generated News Summaries

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Frame In, Frame Out (FIFO) is a new large-scale benchmark designed to measure framing bias in news summaries generated by large language models (LLMs). Grounded in the widely used XSum dataset, FIFO incorporates 15,499 jury-annotated examples alongside 320 expert-labeled instances, achieving an inter-annotator agreement of κ = 0.61 for calibration. Researchers utilized FIFO to analyze 27 distinct summarization models, revealing that LLM-generated summaries frequently exhibit higher calibrated framing rates compared to human-written reference summaries. The study also identified substantial variations in framing bias across different topics and training regimes, noting particularly elevated rates within scientific and public health summaries. This research establishes framing as a significant and often overlooked aspect of summarization quality.

Key takeaway

For NLP Engineers deploying LLMs for news summarization, you must actively integrate framing bias evaluation into your model development and deployment pipelines. Your current LLM-generated summaries likely exhibit higher framing rates than human-written content, particularly in sensitive domains like public health. Prioritize using benchmarks like FIFO to measure and mitigate this bias, ensuring your models produce more neutral and reliable information.

Key insights

LLM-generated news summaries often exhibit significant framing bias, a critical quality dimension measurable with the new FIFO benchmark.

Principles

• Framing bias is a key, overlooked summarization quality metric.
• LLM summaries can show higher framing than human references.
• Framing rates vary significantly across topics and models.

Method

The Frame In, Frame Out (FIFO) benchmark measures framing bias by combining 15,499 jury-annotated examples with 320 expert-labeled instances (κ = 0.61) to validate and calibrate model-based annotations for LLM-generated news summaries.

In practice

• Use benchmarks like FIFO to assess LLM framing bias.
• Scrutinize scientific and public health LLM summaries.
• Analyze training data's effect on framing rates.

Topics

• Framing Bias
• LLM Summarization
• FIFO Benchmark
• News Summaries
• XSum Dataset
• Summarization Quality

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns — LLMs exhibit pervasive gender-stereotypical persuasive language biases across models and languages, reflecting societal norms.
• Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News — Fine-tuned BERT models outperform few-shot LLMs for German climate news framing detection, especially with sentence context.
• Characterizing Narrative Content in Web-scale LLM Pretraining Data — Narrative structure in web-scale LLM pretraining data is measurable and unevenly distributed, impacting model reasoning.
• LLMs for automatic annotation of Mandarin narrative transcripts — LLMs can automate complex discourse-level annotation in non-English speech, significantly reducing time.
• Recency Bias Is Architecture, Not Capability — Recency bias in LLMs is an architectural problem solvable by context window assembly, not model upgrades.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.