How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new analysis extends compression-based memorization techniques to the frozen-base setting, directly measuring, in bits, the information a low-rank adapter writes into a model without altering its base parameters. This research reveals that LoRA adapters store only a few bits per trainable parameter, significantly less than full fine-tuning. The capacity is influenced more by the adapter's placement (e.g., attention vs. MLP layers) than by its parameter count; moving the same parameter budget from attention to MLP nearly doubles storage capacity. Furthermore, removing the frozen base's structure drastically reduces this capacity. Applied to Qwen2.5 fine-tunes, the instrument demonstrates that privacy leakage correlates with the bits an adapter writes, not its nominal parameter count. It also distinguishes between supervised fine-tuning, which copies secrets verbatim, and reinforcement learning, which does not record such information.

Key takeaway

For Machine Learning Engineers designing or evaluating LoRA adapters, you should prioritize adapter placement over raw parameter count to optimize capacity and mitigate privacy risks. Your design choices, such as placing parameters in MLP layers rather than attention, significantly impact storage and potential data leakage. When handling sensitive data, consider reinforcement learning fine-tuning, as it demonstrably avoids verbatim memorization, unlike supervised methods.

Key insights

LoRA adapter capacity and memorization are measurable in bits, revealing placement-dependent storage and privacy implications.

Principles

• Adapter capacity is location-dependent.
• Privacy leakage scales with written bits.
• RL adapters record no verbatim secrets.

Method

Extends compression-based memorization analysis to the frozen-base setting to directly measure, in bits, how much a low-rank adapter writes into a model it never changes.

In practice

• Evaluate adapter placement for capacity.
• Monitor written bits for privacy risk.
• Prefer RL for sensitive data tasks.

Topics

• LoRA Adapters
• Parameter-Efficient Fine-Tuning
• Model Memorization
• Data Privacy
• Reinforcement Learning
• Supervised Fine-tuning

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer

Related on AIssential

• How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning — LoRA adapter capacity and memorization are quantifiable in bits, revealing data leakage risks.
• Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices — Complementary techniques enable LoRA fine-tuning of large LLMs on edge devices by drastically reducing peak memory without quality loss.
• Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA — For closed-book QA with LoRA, data quality is the dominant performance lever once adapter capacity is sufficient.
• Frozen vs Unfrozen Weights: What Adapters Actually Are in LoRA and QLoRA — Frozen weights are static during training; unfrozen weights are updated, defining fine-tuning approaches like LoRA and QLoRA.
• Rethinking Adapter Placement: A Dominant Adaptation Module Perspective — Optimal LoRA adapter placement is concentrated in a single, architecture-dependent, task-stable shallow FFN down-projection.
• What Happens When You Use LoRA On a CNN — LoRA can facilitate some transfer learning in CNNs, but its effectiveness varies significantly with dataset complexity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.