Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

Research investigates internalizing document knowledge directly into a 4-bit Gemma-4-e4b model's LoRA adapter weights for closed-book question answering, eliminating retrieval and context window usage. Across approximately 100 training runs, from single to 99-document corpora, the study found that training-data quality is the primary driver of closed-book accuracy, surpassing LoRA rank, learning rate, and alternative architectures, provided adequate adapter capacity. A single data curation pass, involving shortening gold answers to 1-6 word spans and removing trivia, dramatically increased accuracy from 57.7% to 85.7% on a 15-document corpus. The findings also confirm that LoRA rank must scale with corpus size, and larger ranks necessitate a gentler learning rate. Furthermore, the internalized adapter achieved 84.2% recall, outperforming a BM25-RAG pipeline (58.9%) and a realistic gold-chunk oracle (65.6%) on a 15-document slice with reduced latency.

Key takeaway

For Machine Learning Engineers building closed-book QA systems with LoRA adapters, prioritize meticulous training data curation over complex architectural changes or simply increasing capacity. Your efforts should focus on ensuring gold answers are concise (1-6 words) and removing trivial information, as this yields significant accuracy gains. Systematically scale your LoRA rank with corpus size and remember to adjust your learning rate downwards for larger ranks to prevent performance regressions.

Key insights

Data quality, particularly answer brevity and trivia removal, is the primary lever for closed-book QA accuracy in LoRA adapters.

Principles

• Adequate adapter capacity is foundational.
• Data quality is the dominant lever post-capacity.
• LoRA rank must scale with corpus size.

Method

Train a 4-bit Gemma-4-e4b LoRA adapter (MLP-only, 42 layers) in two stages: CPT-like completion loss, then SFT closed-book QA with answer-token loss.

In practice

• Curate training data for short, canonical answers and remove trivia.
• Scale LoRA rank with corpus size.
• Retune learning rate for higher ranks.

Topics

• LoRA Adapters
• Closed-Book QA
• Training Data Quality
• LoRA Rank Scaling
• Learning Rate Tuning
• Knowledge Internalization

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA — For closed-book QA with LoRA, data quality is the dominant performance lever once adapter capacity is sufficient.
• LoRA, Explained With Spreadsheets: How to Train a Model Without Touching Its Weights — LoRA enables efficient model training by adapting a small, low-rank matrix instead of modifying all original weights.
• How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions — Smaller LoRA-fine-tuned models can achieve near 8B performance for structured extraction, significantly reducing deployment costs.
• The World Of LLMs Post Scaling Laws — AI model development has shifted from pretraining scale to post-training and verifiable data quality due to data exhaustion.
• How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning — LoRA adapter capacity and memorization are measurable in bits, revealing placement-dependent storage and privacy implications.
• Balancing Usefulness and Naturalness: An LLM-based Curation Pipeline for Code Review Comments — Curating LLM training data for code review comments by balancing quality and natural diversity significantly improves model performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.