Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

2026-04-12 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, medium

Summary

A study systematically evaluates Parameter-Efficient Fine-Tuning (PEFT) and quantization techniques on BERTimbau for Brazilian Portuguese Question Answering using the SQuAD-BR dataset. Researchers tested 40 configurations, combining four PEFT methods (LoRA, DoRA, QLoRA, QDoRA) with BERTimbau models of 110M (Base) and 335M (Large) parameters. Key findings indicate that LoRA on BERTimbau-Large achieved 95.8% of baseline performance while reducing training time by 73.5% (F1 of 81.32 vs. 84.86). Higher learning rates (2e-4) significantly boosted PEFT performance, yielding F1 gains up to +19.71 points. Larger models demonstrated twice the quantization resilience, with an F1 loss of 4.83 points compared to 9.56 for smaller models. The research also includes an exploratory evaluation of generative models Tucano and Sabiá, noting their higher GPU memory and training time requirements.

Key takeaway

For AI Engineers developing Question Answering systems for Brazilian Portuguese, prioritizing encoder-based models like BERTimbau with PEFT techniques offers substantial computational efficiency. You should consider LoRA with higher learning rates (2e-4) to achieve competitive performance with significantly reduced training time and GPU memory, making your solutions more sustainable and accessible than large generative LLMs.

Key insights

PEFT methods efficiently fine-tune encoder-based models for low-resource languages, outperforming generative LLMs in resource usage.

Principles

• LoRA reduces training time significantly with minimal performance loss.
• Higher learning rates improve PEFT performance substantially.
• Larger models exhibit greater quantization resilience.

Method

The study systematically evaluated 40 configurations of PEFT methods (LoRA, DoRA, QLoRA, QDoRA) and quantization on BERTimbau (Base/Large) for Question Answering on SQuAD-BR.

In practice

• Use LoRA for efficient fine-tuning of BERTimbau-Large.
• Apply a 2e-4 learning rate for PEFT methods.
• Consider larger models for better quantization robustness.

Topics

• Parameter-Efficient Fine-Tuning
• BERTimbau
• Brazilian Portuguese QA
• LoRA
• Quantization Techniques

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• LLM Fine-tuning: Techniques for Adapting Language Models — Fine-tuning LLMs with PEFT methods like LoRA and QLoRA significantly reduces computational demands while preserving performance.
• LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification — Domain-adapted fine-tuning significantly outperforms general LLMs for specialized legal text classification in Portuguese.
• Specializing a Small Language Model for Closed-Domain Portuguese RAG using Knowledge Graph Supervision — Fine-tuning SLMs with knowledge graph supervision significantly enhances closed-domain RAG performance, surpassing larger LLMs.
• Fine-Tuning Qwen3.5 — Fine-tuning small vision-language models like Qwen3.5-0.8B on domain-specific datasets significantly improves specialized task performance.
• Lispector: Fine-tuning of Language Models for Grammar and Spelling Correction in Brazilian Portuguese — Supervised fine-tuning of smaller models outperforms few-shot prompting for grammar correction in Brazilian Portuguese.
• Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks — Parameter-Efficient Fine-Tuning (PEFT) with adapters and LoRA drastically reduces parameters for instance segmentation while maintaining performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.