Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

2025-12-29 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mechanistic Interpretability · Depth: Expert, extended

Summary

Researchers identified the "Knowing–Using Gap" in LLM fine-tuning, where models memorize new facts but fail to apply them in downstream reasoning tasks. This gap is characterized by an accuracy disparity and a temporal lag between memorization and generalization. To understand this, they developed "self-patching," an intervention technique that monitors internal knowledge permeation. Findings support a "knowledge–circuit misalignment hypothesis," suggesting memorized representations exist but are not routed to computation-effective layers. A simple heuristic strategy, based on effective patch locations, recovered 58–75% of the oracle headroom in generalization failure cases, demonstrating a practical diagnostic finding robust across models, architectures, and knowledge domains like biomedical (STaRK-Prime) and academic (STaRK-MAG).

Key takeaway

For AI Scientists and Machine Learning Engineers fine-tuning LLMs for new knowledge, recognize that memorization does not guarantee generalization. Your models may store facts but fail to route them to reasoning circuits. Consider using diagnostic tools like self-patching to identify knowledge-circuit misalignment. Implementing a simple heuristic, such as patching representations from early or late layers to mid-layers, can recover a significant portion (58–75%) of generalization ability, transforming a routing problem into a tractable alignment challenge.

Key insights

LLMs memorize facts but fail to generalize due to knowledge representations being misaligned with reasoning circuits.

Principles

• The Knowing–Using Gap involves an accuracy gap and temporal lag.
• Knowledge can be stored but not routed to effective computation layers.
• Relocating memorized representations can activate generalization.

Method

Self-patching copies hidden layer representations of an anchor from a source layer to a target layer to measure causal effect on generalization accuracy, scanning all layer pairs.

In practice

• Use self-patching to diagnose knowledge-circuit misalignment.
• Apply a fixed heuristic (early/late to mid-layers) for partial generalization recovery.

Topics

• Large Language Models
• Fine-tuning
• Mechanistic Interpretability
• Knowledge Representation
• Self-patching
• Generalization Failure

Code references

• snap-stanford/stark

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning — LLMs' "Knowing--Using Gap" stems from memorized knowledge being misrouted internally, preventing generalization to reasoning tasks.
• Challenge: Hand coding weights for efficient sequence memorisation — Hand-coding MLP weights for sequence memorization reveals a significant performance gap compared to trained models, despite similar scaling.
• MeMo: Memory as a Model — MeMo integrates new knowledge into LLMs via a dedicated memory model, preserving LLM parameters and preventing forgetting.
• Tracking Representation Dynamics in Large Language Models with Persistent Homology — Persistent homology uncovers internal representation dynamics in LLMs during fine-tuning, showing early topological reorganization and objective-specific trajectories.
• Why Fine-Tuned LLMs (SFT & LoRA) Fail to Reason — Fine-tuned LLMs memorize facts but fail to reason with them due to internal communication breakdowns, not storage.
• Weight Patching: Toward Source-Level Mechanistic Localization in LLMs — Weight Patching identifies where LLM capabilities are encoded in parameters, distinguishing source carriers from aggregation modules.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.