Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

2026-07-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

A study investigates internalizing documents directly into the weights of a 4-bit Gemma-4-e4b model using LoRA adapters to enable closed-book question answering without retrieval or context window budget. Across approximately 100 training runs, ranging from single documents to a 99-document corpus, researchers found that once adapter capacity is sufficient, training-data quality becomes the primary factor influencing closed-book accuracy, surpassing LoRA rank, learning rate, and alternative architectures. A single data curation pass, involving shortening gold answers to 1-6 word spans and removing trivia, significantly improved accuracy from 57.7% to 85.7% on a 15-document corpus. The internalized adapter achieved 84.2% recall, outperforming a BM25-RAG pipeline (58.9%) and a gold-chunk oracle (65.6%) at lower latency. The research also confirms that LoRA rank must increase with corpus size, noting a complex interaction between rank and learning rate. The full debugging process, including three initial misdiagnoses, is presented as an empirical case study.

Key takeaway

For machine learning engineers developing closed-book QA systems with LoRA adapters, prioritize rigorous data curation over extensive architectural experimentation. Your efforts in shortening gold answers to canonical spans and removing trivia can yield significantly higher accuracy, as demonstrated by an 85.7% accuracy with curation versus 57.7% without. Ensure your adapter capacity, specifically LoRA rank, scales appropriately with corpus size, as this forms a critical performance baseline before data quality improvements can take effect.

Key insights

For closed-book QA with LoRA, data quality is the dominant performance lever once adapter capacity is sufficient.

Principles

• Data quality outweighs architectural changes.
• Adapter capacity is a hard performance gate.
• LoRA rank must grow with corpus size.

Method

Documents are internalized into 4-bit Gemma-4-e4b LoRA adapters for closed-book QA. Curation involves shortening gold answers to 1-6 words and dropping trivia.

In practice

• Curate gold answers to 1-6 word spans.
• Remove trivia from training data.
• Increase LoRA rank for larger corpora.

Topics

• LoRA Adapters
• Closed-Book QA
• Data Quality
• Gemma-4-e4b
• Large Language Models
• Model Training Debugging
• Retrieval-Augmented Generation

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA — Data quality, particularly answer brevity and trivia removal, is the primary lever for closed-book QA accuracy in LoRA adapters.
• How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning — LoRA adapter capacity and memorization are quantifiable in bits, revealing data leakage risks.
• Balancing Usefulness and Naturalness: An LLM-based Curation Pipeline for Code Review Comments — Curating LLM training data for code review comments by balancing quality and natural diversity significantly improves model performance.
• How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions — Smaller LoRA-fine-tuned models can achieve near 8B performance for structured extraction, significantly reducing deployment costs.
• When Retrieval Doesn’t Help: A Large-Scale Study of Biomedical RAG — In biomedical QA, RAG offers minimal gains; the LLM's capacity to effectively use retrieved evidence is the primary bottleneck.
• System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5 — Domain-specific datasets and LoRA fine-tuning significantly enhance LLM performance in specialized tasks like classical poetry analysis.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.