I Fine-Tuned One Model 3 Ways: The $50,000 Run Forgot More Than the $1,500 One

2026-06-23 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

An analysis of fine-tuning an 8B model using full fine-tuning, LoRA, and QLoRA revealed a critical trade-off. The most expensive method, full fine-tuning on \$50,000 H100s, achieved a slight win on a target benchmark but significantly degraded the model's general capabilities. This degradation was more pronounced than with cheaper methods like LoRA, which was trained on a \$1,500 RTX 4090. The findings challenge the common perception that full fine-tuning is always the superior "gold standard," demonstrating that while it learns more, it also forgets more. Conversely, LoRA learns less but forgets less, suggesting that the two methods produce structurally different models even when target task performance appears similar.

Key takeaway

For AI engineers and teams managing significant GPU budgets for model fine-tuning, critically re-evaluate the assumption that full fine-tuning is inherently superior. Your expensive H100 runs might achieve target benchmark gains but could silently degrade the model's broader capabilities more than cheaper LoRA or QLoRA approaches. Prioritize comprehensive evaluation of general ability alongside task-specific metrics to avoid costly regressions.

Key insights

Expensive full fine-tuning can degrade a model's general ability more than cheaper parameter-efficient methods.

Principles

• Full fine-tuning learns more and forgets more.
• LoRA learns less and forgets less.
• LoRA and full fine-tuning yield structurally different models.

In practice

• Reproduce cheap fine-tuning on a single consumer GPU.
• Evaluate model general ability beyond target benchmarks.

Topics

• Fine-tuning
• LoRA
• QLoRA
• Model Degradation
• GPU Performance
• Parameter-Efficient Fine-Tuning

Best for: Machine Learning Engineer, AI Engineer, Director of AI/ML

Related on AIssential

• The Tiny Idea That Lets Anyone Fine-Tune AI — Low-rank decomposition enables efficient fine-tuning by adding small, task-specific corrections to large pre-trained models.
• Fine-Tuning a 7B LLM Required 4 A100s. LoRA Did It on One GPU. Here Is the Math Behind Why. — Fine-tuning LLMs involves low-dimensional weight updates, enabling efficient adaptation via LoRA and QLoRA.
• Gemma 4 Fine-Tuning on Single GPU | Training Gemma 4 With Unsloth on Custom Dataset (🔴 Live) — Fine-tuning LLMs may offer better accuracy and lower GPU RAM than prompting alone.
• How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions — Smaller LoRA-fine-tuned models can achieve near 8B performance for structured extraction, significantly reducing deployment costs.
• LLM Fine-tuning: Techniques for Adapting Language Models — Fine-tuning LLMs with PEFT methods like LoRA and QLoRA significantly reduces computational demands while preserving performance.
• How to Fine-Tune an LLM: SFT, LoRA, QLoRA and DPO Explained — Fine-tuning adapts LLMs for specific tasks and behaviors using SFT, LoRA, QLoRA, and DPO.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.