Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh

2026-07-26 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

New research explored whether fine-tuning small language models on legal examples with relevant statutory provisions improves their performance on legal question-answering tasks. Researchers curated 2,165 bilingual QA records from six Bangladeshi acts and three schedules, then fine-tuned Qwen3.5 models at 0.8B, 2B, and 4B parameters. Evaluation used the 2022 and 2023 Bangladesh Bar Council exams in both Bangla and machine-translated English. The 0.8B model's 2022 English FAISS score significantly increased from 2 to 34 out of 100 after fine-tuning. Gains were statistically significant for the 0.8B and 2B models, though the 4B model showed no net gain, with English conditions regressing despite Bangla improvements. Fine-tuning also dramatically reduced language drift from 44.0-53.2% to 0.2-0.7% (adjusted $p<.001$). This indicates that retrieval quality is not the sole bottleneck for small bilingual legal models, which also vary in their use of supplied law and language consistency. The dataset is publicly available.

Key takeaway

For NLP Engineers developing legal QA systems, especially in bilingual or low-resource settings, consider context-injected fine-tuning for small language models. Your efforts on models like Qwen3.5 at 0.8B or 2B parameters can significantly improve their ability to apply legal provisions and maintain consistent output language. Do not assume larger models will automatically yield better results from this approach; focus on targeted fine-tuning for specific performance gains and reduced language drift.

Key insights

Fine-tuning small LLMs with legal context improves their ability to use provided law and maintain language consistency in legal QA.

Principles

• Small LLMs benefit from context-injected fine-tuning.
• Language consistency improves with targeted fine-tuning.
• Model scale doesn't guarantee fine-tuning gains.

Method

Curate bilingual legal QA records, fine-tune small LLMs (0.8B-2B) on examples containing relevant law, and evaluate on standardized legal exams for performance and language consistency.

In practice

• Use context-injected fine-tuning for legal QA.
• Prioritize smaller LLMs (0.8B, 2B) for legal fine-tuning.
• Fine-tune to reduce language drift in bilingual models.

Topics

• Legal Question Answering
• Small Language Models
• Fine-tuning
• Bilingual NLP
• Bangladeshi Law
• Qwen3.5

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Improving Answer Extraction in Context-based Question Answering Systems Using LLMs — Fine-tuning pre-trained LLMs on benchmark QA datasets significantly improves answer extraction accuracy and contextual understanding in QA systems.
• Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning — Reinforcement learning with verifiable rewards significantly enhances small language models for legal NMT, outperforming supervised fine-tuning.
• bLLeQA: Benchmarking LLMs for Grounded Legal Question-Answering in French and Dutch — RAG systems for legal Q&A in French and Dutch face significant challenges, especially in generation quality and source refusal.
• FineREX: Fine-Tuned NER-RE for Human Smuggling Knowledge Graphs — Domain-specific fine-tuning of LLMs substantially improves knowledge graph quality and efficiency for complex legal text.
• Asking the Right Questions: Can expert-prompted LLMs reformulate legal queries from non-experts? — LLMs can reformulate non-expert legal queries, improving lawyer-client communication, but require careful safety consideration.
• TW-LegalBench: Measuring Taiwanese Legal Understanding — LLMs demonstrate proficiency in legal qualification exams but struggle with precise statutory citation and advanced reasoning in civil-law contexts.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.