How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning

2026-07-23 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

A new study measures the data capacity and memorization of LoRA adapters in parameter-efficient fine-tuning, challenging the view that adapters solely represent skills rather than data records. Researchers extended compression-based memorization analysis to the frozen-base setting, quantifying in bits how much an adapter "writes" into a model without altering its base. Findings indicate adapters store only a couple of bits per trainable parameter, significantly less than full fine-tuning. This capacity is influenced by parameter placement; moving parameters from attention to the MLP layer nearly doubles storage. Furthermore, removing the frozen base's structure drastically reduces capacity. When applied to Qwen2.5 fine-tunes, privacy leakage correlated with the actual bits an adapter writes, not its nominal parameter count. The analysis also distinguished between supervised fine-tuning, which copies secrets verbatim, and reinforcement learning adapters, which do not record such information when trained on verifiable rewards. This method provides a quantifiable approach to design against memorization.

Key takeaway

For AI Security Engineers evaluating fine-tuned models, you should prioritize measuring the actual bits an adapter writes rather than solely relying on its parameter count to assess privacy leakage. Your risk assessment for supervised fine-tuning must account for verbatim data memorization, which is less prevalent in reinforcement learning-trained adapters. Consider adjusting adapter parameter placement, such as favoring MLP layers, to optimize capacity while managing memorization risks.

Key insights

LoRA adapter capacity and memorization are quantifiable in bits, revealing data leakage risks.

Principles

• Adapter capacity depends on parameter location, not just count.
• Base model structure is critical for adapter capacity.
• RL-trained adapters show less verbatim memorization than SFT.

Method

Extended compression-based memorization analysis to frozen-base models to measure bits written by adapters.

In practice

• Evaluate adapter memorization by measuring bits written.
• Prioritize MLP over attention for higher capacity needs.
• Use RL fine-tuning to mitigate verbatim data leakage.

Topics

• LoRA Adapters
• Parameter-Efficient Fine-Tuning
• Model Memorization
• Data Leakage
• Qwen2.5
• Reinforcement Learning

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning — LoRA adapter capacity and memorization are measurable in bits, revealing placement-dependent storage and privacy implications.
• The Tiny Idea That Lets Anyone Fine-Tune AI — Low-rank decomposition enables efficient fine-tuning by adding small, task-specific corrections to large pre-trained models.
• Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA — For closed-book QA with LoRA, data quality is the dominant performance lever once adapter capacity is sufficient.
• How LoRA Remembers? A Parametric Memory Law for LLM Finetuning — The Parametric Memory Law quantifies LoRA's exact memory capacity in LLMs, linking loss reduction to effective parameters and sequence length.
• Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters — Finer parameter steps in PEFT, like CP tensor adapters, diagnose budget sensitivity but don't always improve accuracy-budget trade-offs over LoRA.
• Frozen vs Unfrozen Weights: What Adapters Actually Are in LoRA and QLoRA — Frozen weights are static during training; unfrozen weights are updated, defining fine-tuning approaches like LoRA and QLoRA.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.