LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs

2026-06-04 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Data Science & Analytics · Depth: Advanced, quick

Summary

The PropMe framework introduces a propensity-aware approach for evaluating large language model memorization, distinguishing between "capability attacks" that force data reproduction and "propensity" for ordinary leakage. This framework, utilizing a metric transformation and the SimpleTrace pipeline built on infini-gram, deterministically attributes model generations to training corpora like Common Pile and Dynaword. Evaluating Comma and DFM Decoder models, the study found a consistent gap: prefix attacks elicited substantially stronger memorization signals than generic prompts, yet overall propensity scores remained low. This indicates models can reveal training data when directly elicited but rarely do so in non-adversarial settings. Furthermore, DFM Decoder, continually pre-trained from Comma, exhibited reduced memorization for Common Pile, suggesting that emphasizing partially different data during later training can decrease memorization capability. The research encourages comprehensive memorization audits reporting both worst-case extractability and ordinary leakage propensity.

Key takeaway

For AI Security Engineers evaluating LLM data leakage, you must differentiate between a model's potential to leak and its actual propensity under normal use. Your audits should report both worst-case extractability, often revealed by prefix attacks, and ordinary leakage propensity. This comprehensive view helps you accurately assess privacy risks. When designing continual pre-training, emphasize partially different data. This can actively reduce memorization capability, as DFM Decoder showed for Common Pile.

Key insights

LLMs can be forced to leak training data, but rarely do so under ordinary, non-adversarial use.

Principles

• Memorization capability differs from leakage propensity.
• Prefix attacks elicit stronger memorization signals.
• Continual pre-training can reduce memorization.

Method

PropMe framework contrasts prefix-based capability attacks with non-adversarial evaluations, using a metric transformation and SimpleTrace (built on infini-gram) to attribute generations and compute verbatim, near-verbatim, and propensity metrics.

In practice

• Audit LLMs for both worst-case extractability.
• Report ordinary leakage propensity in audits.
• Consider diverse data for continual pre-training.

Topics

• LLM Memorization
• Data Leakage
• PropMe Framework
• SimpleTrace
• Continual Pre-training
• AI Security Audits

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs — LLMs can leak training data under adversarial prompts but rarely do so in ordinary, non-adversarial use.
• Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects — DLMs exhibit lower PII leakage than ARMs, but higher sampling resolution increases memorization probability.
• Pretraining Data Filtering for Open-Weight AI Safety — Pretraining data filtering effectively builds tamper-resistant safeguards into open-weight LLMs by preventing undesirable knowledge acquisition.
• Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data — LLMs can learn dictionary encodings in-context, enabling lossless prompt compression without fine-tuning and preserving analytical accuracy.
• On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance — LLM annotation reliability hinges on definition alignment, not just text memorization, as prompt-based corrections are largely ineffective.
• Did the Model See the Benchmark During Training? Detecting LLM Contamination — A lightweight method estimates LLM benchmark contamination by analyzing model behavior when training data is unknown.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.