Evaluating Open-Source LLMs for Text Summarization and Named Entity Recognition in Long, Unstructured Text

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

An evaluation of open-source Large Language Models (LLMs) investigates their utility for abstractive summarization and fine-grained Named Entity Recognition (NER) in long, unstructured historical documents. Focusing on witness testimonies from the South African Truth and Reconciliation Commission (TRC), which detail human rights violations during apartheid, the study assessed LLMs in zero-shot settings. Results indicate strong summarization performance, with most LLMs achieving a maximum BERTScore of 0.77, outperforming non-LLM baselines. However, NER performance remained limited, peaking at an F1-score of 0.61. The analysis suggests LLMs prioritize stylistic fluency over factual precision. A two-stage pipeline, involving summarization followed by NER on the generated summaries, demonstrated measurable improvements in overall performance, enhancing accessibility for these historically significant but difficult-to-index texts.

Key takeaway

For NLP Engineers working with long, unstructured historical documents, you should consider a two-stage pipeline: first, use open-source LLMs for abstractive summarization, leveraging their strong performance (BERTScore 0.77). Subsequently, apply NER on these generated summaries to mitigate the LLMs' current limitations in factual precision (F1-score 0.61) for entity extraction. This approach can significantly improve accessibility and indexing of challenging texts like historical testimonies, despite the inherent fluency-precision trade-off.

Key insights

Open-source LLMs excel at summarization but struggle with factual NER in historical documents, suggesting a fluency-precision trade-off.

Principles

• LLMs prioritize fluency over factual precision.
• Zero-shot LLM NER is currently limited.
• Summarization can precede NER for improvement.

Method

A two-stage pipeline first performs abstractive summarization using LLMs, then applies Named Entity Recognition on the generated summaries to improve overall performance on long, unstructured texts.

In practice

• Apply LLMs for abstractive summarization first.
• Use a two-stage pipeline for NER on summaries.
• Consider LLM fluency-precision trade-offs.

Topics

• Open-source LLMs
• Text Summarization
• Named Entity Recognition
• Historical Documents
• Zero-shot Learning
• Apartheid Archives

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Data Augmentation for Historical NER: A Systematic Comparison of Lexical and LLM-based Approaches — This paper systematically compares lexical and LLM-based data augmentation for Historical Named Entity Recognition.
• A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization — OmniCSEval offers a robust, multi-dimensional benchmark for evaluating LLMs in conversation summarization across diverse real-world scenarios.
• Using a Local LLM as a Zero-Shot Classifier — Locally hosted LLMs can effectively perform zero-shot classification on short, semantically complex texts where traditional methods fail.
• Semantic marking of named entities in The Lusiads — Hybrid NER workflows improve accuracy for historical literary texts, overcoming limitations of models trained on modern corpora.
• A BART-based approach with hierarchical strategy for Vietnamese abstractive multi-document summarization — A BART-based hierarchical strategy with golden summary-driven shortening improves Vietnamese multi-document abstractive summarization, achieving 0.2468 ROUGE2-F1.
• Training a Named Entity Recognition Model with Prodigy and Transfer Learning — Efficient NER model training for specific domains is achieved through semi-automatic annotation and transfer learning with tools like Prodigy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.