How to Fine-Tune an LLM: SFT, LoRA, QLoRA and DPO Explained

2026-05-17 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, extended

Summary

Fine-tuning is a critical technique for adapting Large Language Models (LLMs) to specific tasks like coding or persona emulation, moving beyond their initial next-token prediction capabilities. This process enhances domain-specific knowledge and instruction following. The article differentiates fine-tuning from Retrieval Augmented Generation (RAG), noting that fine-tuning alters model behavior and reasoning, while RAG is better for external or frequently changing knowledge. It details Supervised Fine-Tuning (SFT) using instruction-output pairs and various dataset formats (Alpaca, ShareGPT, OpenAI Chat, Dolly, FLAN). The post also explores Parameter Efficient Fine-Tuning (PEFT) methods, specifically Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA), which significantly reduce computational resources by training only small adapter weights on a frozen or 4-bit quantized base model. Finally, it covers Preference Alignment techniques like Direct Preference Optimization (DPO), which trains models to favor specific responses over generic ones, building on an SFT-tuned model.

Key takeaway

For AI Engineers developing specialized LLMs, understanding the nuances of fine-tuning methods is crucial. If your goal is to alter an LLM's reasoning or persona, prioritize SFT and DPO. For resource-constrained environments, QLoRA with tools like Unsloth offers a practical path to fine-tune large models like Llama 3.1 8B on a single GPU, enabling efficient deployment of custom-behaved LLMs.

Key insights

Fine-tuning adapts LLMs for specific tasks and behaviors using SFT, LoRA, QLoRA, and DPO.

Principles

• Fine-tuning changes model behavior; RAG updates knowledge.
• LoRA/QLoRA train small adapters, preserving base model weights.
• DPO aligns model output with preferred responses.

Method

SFT involves training on instruction-output pairs. LoRA/QLoRA add and train small adapter matrices (A, B) to a frozen or 4-bit quantized base model. DPO optimizes a model to assign higher probability to chosen responses over rejected ones.

In practice

• Use Unsloth for efficient Llama 3.1 8B QLoRA fine-tuning.
• Apply LoRA to attention and MLP layers for persona tasks.
• Generate synthetic datasets with strong LLMs like Claude.

Topics

• LLM Fine-Tuning
• Supervised Fine-Tuning
• Parameter Efficient Fine-Tuning
• LoRA
• QLoRA

Code references

• VrityaCodeRishi/Vritya-Tiny-163M-1

Best for: Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• LoRA & QLoRA Mastery: The Beginner-to-Advanced Guide to Efficient LLM Fine-Tuning — LoRA and QLoRA enable efficient LLM fine-tuning by adapting a small fraction of parameters, drastically reducing memory and compute.
• Understanding post-training of LLMs: SFT — Post-training guides LLM behavior and expression, not knowledge, primarily via SFT and preference optimization.
• Preference Fine-Tuning LFM 2 Using DPO — LFM 2 models combine efficient architecture with DPO fine-tuning for high-performance, edge-deployable SLMs.
• Have you tried Axolotl for LLM Fine-Tuning? — Axolotl simplifies LLM fine-tuning by integrating diverse methods and models into a unified, configurable framework.
• LLM Fine-tuning: Techniques for Adapting Language Models — Fine-tuning LLMs with PEFT methods like LoRA and QLoRA significantly reduces computational demands while preserving performance.
• The Evolution of Adaptation: From Full Fine-Tuning to LoRA Mastery — LoRA and QLoRA democratize LLM fine-tuning by drastically reducing computational and memory requirements.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.