Let's reproduce GPT-2 (124M)

2024-06-09 · Source: Andrej Karpathy · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, extended

Summary

This content details the reproduction and optimization of the GPT-2 124 million parameter model, building it from scratch using PyTorch. It covers the architectural implementation, including Transformer blocks, MLP, and attention mechanisms, and explains how to load and verify pre-trained weights from Hugging Face Transformers. The analysis then shifts to optimizing the training process, discussing techniques like weight tying, careful initialization, mixed-precision training (TF32, BF16), and `torch.compile` for performance gains. It also delves into algorithmic improvements, such as AdamW optimizer settings, gradient clipping, learning rate schedules, and gradient accumulation for distributed training across multiple GPUs using PyTorch DDP. Finally, the content addresses dataset upgrades to FineWeb-Edu, evaluation using validation loss and HellaSwag accuracy, and checkpointing, demonstrating how to achieve and surpass GPT-2 124M performance with significantly fewer training tokens.

Key takeaway

For Deep Learning Engineers aiming to optimize large language model training, adopting modern PyTorch features like `torch.compile`, mixed-precision training (BF16), and PyTorch DDP is crucial. You should also meticulously align with established hyperparameter settings, such as AdamW betas and learning rate schedules, and consider algorithmic optimizations like weight tying and gradient accumulation to maximize GPU utilization and achieve competitive model performance efficiently.

Key insights

Reproducing and optimizing GPT-2 from scratch reveals significant performance gains through careful implementation and modern PyTorch features.

Principles

• Memory access patterns dictate GPU performance more than raw FLOPs.
• Weight tying and specific initialization improve model efficiency and stability.
• Distributed training requires careful data and gradient synchronization.

Method

Implement GPT-2 architecture in PyTorch, load pre-trained weights for verification, then apply performance optimizations like mixed precision, `torch.compile`, and DDP, while carefully managing hyperparameters and data loading for efficient training.

In practice

• Use `torch.set_float32_matmul_precision('high')` for TF32 acceleration.
• Wrap model in `torch.amp.autocast(dtype=torch.bfloat16)` for BF16 mixed precision.
• Employ `torch.compile(model)` to reduce Python overhead and fuse kernels.

Topics

• GPT-2 Reproduction
• Transformer Architecture
• Distributed Data Parallel
• Mixed Precision Training
• Flash Attention

Best for: Deep Learning Engineer, Machine Learning Engineer, AI Researcher

Related on AIssential

• AI in Multiple GPUs: ZeRO & FSDP — ZeRO optimizes distributed AI training by sharding model states across GPUs, drastically reducing memory footprint.
• Decoder only Transformer : Building a GPT-2 model prototype to make it understand Natural Language… — Training a GPT-2 prototype on classic novels demonstrates decoder-only Transformer architecture and performance benefits of pre-trained weights.
• Why build a bigger model when you can just loop twice for twice the power? — Parallel Loop Transformers enable simultaneous model refinement, but empirical data shows two loops are often optimal, with more leading to performance regression.
• Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding — DAS accelerates RL rollouts by adapting a training-free drafter and length-aware scheduling to mitigate the long-tail bottleneck.
• GPT 5.5 Arrives, DeepSeek V4 Drops, and the Compute War Intensifies — Domain-specific training and cost-efficiency are becoming critical differentiators for new large language models amidst compute scarcity.
• Zyphra Introduces Tensor and Sequence Parallelism (TSP): A Hardware-Aware Training and Inference Strategy That Delivers 2.6x Throughput Over Matched TP+SP Baselines — TSP simultaneously shards model weights and token sequences, resolving GPU memory bottlenecks for long-context transformers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Andrej Karpathy.