Build an LLM from Scratch 7: Instruction Finetuning

2025-04-11 · Source: Sebastian Raschka · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Intermediate, extended

Summary

This content introduces instruction fine-tuning for Large Language Models (LLMs), building a personal assistant capable of free-form text responses. It details the process of preparing a small, custom dataset of 1,100 training examples in Alpaca format, converting JSON entries into a prompt-style template, and tokenizing the data. The process involves padding data on a batch basis to optimize training efficiency and masking padding tokens in the loss function. The author uses a GPT-2 medium model with 355 million parameters, a step up from previous chapters' 124 million parameter model, to achieve better performance on a MacBook Air. The fine-tuned model demonstrates improved instruction following, such as converting active to passive voice, compared to its pre-trained state. The content also explores evaluating LLMs using another LLM, specifically Ollama, to score responses, and discusses advanced topics like Direct Preference Optimization (DPO) for aligning models to human preferences.

Key takeaway

For AI engineers developing custom LLM applications, understanding instruction fine-tuning is crucial. You should focus on meticulously preparing instruction datasets, optimizing data loading with batch-specific padding and masked loss, and leveraging tools like Ollama for automated evaluation. This approach allows you to adapt pre-trained models for specific tasks, ensuring more flexible and accurate free-form text generation, even with limited computational resources.

Key insights

Instruction fine-tuning enables LLMs to follow diverse commands and generate free-form text responses.

Principles

• Instruction fine-tuning reuses next-token prediction loss.
• Padding should be batch-specific for efficiency.
• Mask padding tokens to prevent loss function influence.

Method

Prepare instruction data in a prompt-style template, tokenize, pad on a batch basis, mask padding tokens, and then fine-tune a pre-trained LLM using next-token prediction.

In practice

• Use Alpaca format for instruction datasets.
• Employ `ticktoken` for efficient tokenization.
• Evaluate LLM responses using another LLM (e.g., Ollama).

Topics

• Instruction Fine-tuning
• LLM Data Preparation
• Prompt Templating
• Ollama LLM Evaluation
• Direct Preference Optimization

Best for: AI Student, Machine Learning Engineer, AI Engineer

Related on AIssential

• Developing an LLM: Building, Training, Finetuning — LLM development involves building architecture, pre-training on vast datasets, and fine-tuning for specific tasks and behaviors.
• LLM Instruction Tuning & DPO via H2O Enterprise LLM Studio | Part 13 — Instruction tuning enhances frontier models for specific domains, improving cost-efficiency and performance with labeled data.
• Is Fine-Tuning Still Valuable? — Effective LLM fine-tuning requires robust evaluation systems and iterative data curation, especially for domain-specific tasks.
• 6 Things I Learned Building LLMs From Scratch That No Tutorial Teaches You — LLM architecture involves non-obvious design choices significantly impacting performance, cost, and training stability.
• Build an LLM from Scratch 6: Finetuning for Classification — Fine-tuning pre-trained LLMs for specific classification tasks is efficient and effective, even with smaller models.
• System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5 — Domain-specific datasets and LoRA fine-tuning significantly enhance LLM performance in specialized tasks like classical poetry analysis.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Sebastian Raschka.