Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Knowledgeless Language Models (KLLMs) represent a novel training paradigm designed to reduce large language models' reliance on parametric factual knowledge. These models are pretrained on corpora where named entities are systematically anonymized, removing a primary channel for entity-linked factual supervision. This intervention significantly diminishes closed-book factual recall while enhancing performance on tasks requiring evidence-grounded reasoning. Across various model scales, KLLMs consistently surpass standard language models on benchmarks for contextual question answering, fact verification, and hallucination detection. Notably, they achieve up to 20–25% relative gains in retrieval-grounded scenarios with imperfect evidence and demonstrate improved calibration metrics like ECE, Brier score, and AUROC, alongside more reliable abstention behavior. This approach suggests pretraining-time control over knowledge acquisition can create more evidence-sensitive base models.

Key takeaway

For machine learning engineers building retrieval-augmented generation (RAG) systems, adopting Knowledgeless Language Models (KLLMs) during pretraining can significantly enhance model reliability. Your models will exhibit reduced hallucination and improved factual grounding by relying more on provided context, especially with imperfect evidence. This approach offers a robust base model, complementing existing RAG and tool-based systems, leading to better calibrated predictions and more reliable abstention behavior in critical applications.

Key insights

Suppressing entity-linked supervision during pretraining shifts LLMs from parametric recall to evidence-grounded reasoning, improving reliability.

Principles

• Pretraining signal fundamentally shapes LLM epistemic behavior.
• Suppressing entity-linked supervision enhances evidence reliance.
• KLLMs improve robustness to imperfect evidence and calibration.

Method

Named entities in pretraining corpora are anonymized with placeholder tokens (e.g., "PERSON184"). Models are trained from scratch on this data. Inference involves anonymizing inputs, processing, then de-anonymizing outputs.

In practice

• Pretrain LLMs on entity-anonymized corpora.
• Integrate KLLMs into RAG for improved robustness.
• Use KLLMs for fact verification and hallucination detection.

Topics

• Knowledgeless Language Models
• Parametric Knowledge Suppression
• Evidence-Grounded Reasoning
• Retrieval-Augmented Generation
• LLM Hallucination
• Model Calibration
• Named Entity Anonymization

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling — Anonymizing named entities during pretraining shifts LLMs from parametric recall to evidence-grounded reasoning.
• Büyük Dil Modellerinin Hafızasını Güçlendiren Yaklaşım:Retrieval-Augmented Generation (RAG) — RAG enhances LLMs by providing external, real-time context, reducing hallucinations and improving factual accuracy.
• Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies — Formulating factuality checking as a true/false reading comprehension task with test-taking strategies improves LLM efficiency and accuracy.
• Evidence-Augmented Generation Reasoning for Extremely Low-Resource Language Decipherment — Extremely low-resource language decipherment benefits from dynamic rule discovery via LLM-generated examples and semantic retrieval.
• AI Doesn’t Search. It Remembers. — LLMs remember knowledge through trained weights, not by searching, akin to human semantic memory.
• FIDES: Faithful Inference via Deep Evidence Signals for Retrieval-Memory Conflict in RAG — FIDES resolves RAG retrieval-memory conflict by selectively applying contrastive decoding based on token-level conflict concentration.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.