Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling

2026-07-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Knowledge--"Less" Language Models (KLLMs) represent a new epistemic training paradigm designed to reduce language models' reliance on internal parametric knowledge. Introduced in a recent study, KLLMs are pretrained on corpora where named entities are anonymized, effectively removing a primary source of entity-linked factual supervision. This intervention significantly decreases closed-book factual recall while enhancing performance on tasks where relevant information is supplied as context. Across various model scales, KLLMs consistently surpassed matched baselines in contextual question answering, fact verification, and hallucination detection benchmarks. Notably, in retrieval-grounded scenarios involving imperfect evidence, KLLMs demonstrated improved robustness, achieving up to 20-25% relative gains compared to standard language models. They also exhibited better calibration, with improved ECE, Brier score, and AUROC, alongside more reliable abstention behavior. This approach suggests that controlling knowledge acquisition during pretraining can augment retrieval-augmented and tool-based systems by providing a more evidence-sensitive base model.

Key takeaway

For Machine Learning Engineers developing robust, evidence-grounded LLM applications, consider integrating pretraining techniques that suppress parametric recall. Your models will exhibit improved reliability and up to 20-25% relative gains in retrieval-grounded settings, especially with imperfect evidence. This approach enhances calibration and abstention behavior, making your systems more trustworthy for critical tasks like fact verification and hallucination detection.

Key insights

Anonymizing named entities during pretraining shifts LLMs from parametric recall to evidence-grounded reasoning.

Principles

• Suppressing entity-linked supervision improves evidence reliance.
• Pretraining knowledge control complements RAG systems.
• Reduced parametric recall enhances robustness with imperfect evidence.

Method

Pretrain language models on corpora with anonymized named entities to remove entity-linked factual supervision, thereby reducing parametric recall and promoting reliance on external evidence.

In practice

• Use KLLM-like pretraining for evidence-sensitive base models.
• Apply KLLMs in retrieval-grounded QA with noisy evidence.
• Deploy KLLMs for improved hallucination detection.

Topics

• Knowledge--'Less' Language Models
• Evidence-Grounded Reasoning
• Parametric Knowledge Suppression
• Retrieval-Augmented Generation
• Contextual Question Answering
• Hallucination Detection

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling — Suppressing entity-linked supervision during pretraining shifts LLMs from parametric recall to evidence-grounded reasoning, improving reliability.
• Thinking to recall: How reasoning unlocks parametric knowledge in LLMs — Reasoning in LLMs unlocks parametric knowledge through latent computation and factual priming, even for simple facts.
• Pretraining Data Filtering for Open-Weight AI Safety — Pretraining data filtering effectively builds tamper-resistant safeguards into open-weight LLMs by preventing undesirable knowledge acquisition.
• Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining — Language models can naturally "ungrok" learned rules during pretraining, dictated by the rule's support frequency in the corpus.
• It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief — LLMs' context-following behavior is significantly influenced by the linguistic framing of user beliefs, not just the content.
• Evidence-Augmented Generation Reasoning for Extremely Low-Resource Language Decipherment — Extremely low-resource language decipherment benefits from dynamic rule discovery via LLM-generated examples and semantic retrieval.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.