Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms

2026-04-23 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new dataset called RedirectQA has been introduced to analyze non-verbatim memorization in large language models (LLMs), specifically focusing on how entity surface forms influence factual recall. Unlike previous methods that used single canonical names, RedirectQA leverages Wikipedia redirect information to link Wikidata factual triples with various surface forms, including aliases, abbreviations, spelling variants, and common errors. Researchers evaluated 13 LLMs using this dataset and found that prediction outcomes frequently change based solely on the entity's surface form. This inconsistency varies by category, with models showing more robustness to minor orthographic changes than to significant lexical variations like aliases. Frequency analysis indicates that both entity-level and surface-level frequencies correlate with accuracy, with entity frequency often having an impact beyond surface frequency, suggesting factual memorization is neither entirely surface-specific nor fully surface-invariant.

Key takeaway

For research scientists evaluating LLM reliability, you should incorporate diverse entity surface forms, including aliases and common errors, into your QA datasets. Relying solely on canonical names can obscure inconsistencies in factual recall, as models demonstrate varying robustness to different lexical and orthographic variations. Expanding your evaluation to include RedirectQA-like methodologies will provide a more accurate assessment of an LLM's true factual knowledge and its limitations.

Key insights

LLM factual recall is sensitive to entity surface forms, highlighting the need for diverse evaluation methods.

Principles

• Factual memorization is not purely surface-specific.
• Entity frequency impacts recall beyond surface frequency.

Method

RedirectQA uses Wikipedia redirects to associate Wikidata triples with categorized entity surface forms for evaluating LLM non-verbatim memorization.

In practice

• Test LLMs with diverse entity surface forms.
• Include abbreviations and aliases in test sets.

Topics

• Large Language Models
• Non-Verbatim Memorization
• Entity Surface Forms
• RedirectQA Dataset
• Factual Knowledge Evaluation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency — Factual recall in LLMs scales predictably with model size and topic frequency in training data.
• Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluation — Lexical faithfulness in LLM evaluation misclassifies valid clinical reasoning as hallucination, inflating error rates.
• On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance — LLM annotation reliability hinges on definition alignment, not just text memorization, as prompt-based corrections are largely ineffective.
• Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models? — Factual recall mechanisms in multimodal SLMs are weaker and less localized for speech inputs compared to text.
• Substrate Asymmetry in User-Side Memory: A Diagnostic Framework — User-side LLM memory has orthogonal axes, revealing substrate-specific failures hidden by aggregate personalization metrics.
• Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs — Reasoning in LLMs improves single-hop factual recall through computational buffering and factual priming, despite inherent hallucination risks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.