LoRA, Explained With Spreadsheets: How to Train a Model Without Touching Its Weights

2026-07-26 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, long

Summary

Low-Rank Adaptation (LoRA) is a technique enabling the training of large language models on resource-constrained hardware, such as a 6GB laptop GPU, by significantly reducing the number of trainable parameters. Instead of updating all original model weights, LoRA freezes them and introduces a small set of new, trainable weights. For a model with 494 million parameters, LoRA can reduce the trainable count to as low as 270,336 (0.055% with rank r=4) or up to 2,162,688 (0.438% with r=32). The method involves adding a low-rank matrix (B @ A) as a correction layer to the frozen weights, controlled by parameters like r (size of A and B), lora_alpha (correction strength), target_modules (which layers to adapt), and lora_dropout (regularization). A demonstration showed a model learning a specific output pattern but also exhibiting factual inaccuracies due to narrow training data.

Key takeaway

For AI engineers fine-tuning large models on resource-limited GPUs, LoRA offers a critical solution. You can train models with hundreds of millions of parameters by updating less than 1% of them, drastically cutting memory and computational needs. Be mindful that highly specific or small LoRA datasets might inadvertently degrade general knowledge, so validate model behavior broadly after training. Consider adjusting "r" and "lora_alpha" to optimize performance and resource usage for your specific task.

Key insights

LoRA enables efficient model training by adapting a small, low-rank matrix instead of modifying all original weights.

Principles

• LoRA freezes original model weights, training only small additive matrices.
• Low-rank adaptation significantly reduces trainable parameters for fine-tuning.
• Narrow LoRA training can inadvertently alter unrelated model knowledge.

Method

LoRA attaches small, trainable A and B matrices to specific model layers. Their product (B @ A) forms a correction layer added to frozen original weights. Only A and B are updated during training.

In practice

• Use the "peft" library for LoRA implementation.
• Adjust "r" to balance detail learning with memory/speed.
• Set "lora_alpha" to scale correction strength post-training.

Topics

• Low-Rank Adaptation
• Parameter Efficient Fine-Tuning
• Large Language Models
• GPU Memory Optimization
• Model Fine-tuning
• Deep Learning Training

Code references

• NehaKhann/ai-engineering-journey

Best for: AI Student, Machine Learning Engineer, AI Engineer

Related on AIssential

• LoRA, Explained With Spreadsheets: How to Train a Model Without Touching Its Weights — LoRA enables efficient large model training by updating only a tiny fraction of parameters, significantly reducing memory needs.
• What is LoRA actually? — LoRA efficiently adapts large models by learning low-rank weight updates, drastically reducing parameters without sacrificing quality or adding latency.
• The Tiny Idea That Lets Anyone Fine-Tune AI — Low-rank decomposition enables efficient fine-tuning by adding small, task-specific corrections to large pre-trained models.
• Super Weights in LLMs and the Failure of Selective Training — Parameter importance does not imply trainability; effective LLM fine-tuning requires structured, layer-wide updates.
• What Happens When You Use LoRA On a CNN — LoRA can facilitate some transfer learning in CNNs, but its effectiveness varies significantly with dataset complexity.
• LoRA: Understanding Low-Rank Adaptation — LoRA approximates full weight updates using low-rank matrices for efficient, latency-free fine-tuning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.