LLMs: A Journey Through Time and Architecture

2024-09-24 · Source: Sebastian Raschka · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

This analysis details the evolution of large language models (LLMs) from GPT-1 in 2018 to Llama 3.1 in 2024, focusing on changes in model size, training datasets, pre-training pipelines, and architectural innovations. GPT-1 started with 124 million parameters, while Llama 3 now ranges from 8 billion to 45 billion. Training datasets have expanded dramatically, from GPT-2's 40 billion tokens to Llama 3's 15 trillion tokens, with increased emphasis on data filtering, mixing, and synthesis. Pre-training pipelines have evolved into multi-stage procedures, incorporating long-context pre-training and high-quality data annealing. Architectural tweaks, such as replacing LayerNorm with RMSNorm, absolute positional embeddings with RoPE, and masked multi-head attention with grouped query attention (GQA), have improved efficiency. Other innovations include Mixture of Experts (MoE) in models like Mixtral and sliding window attention in Gemma 2, aiming to enhance performance and computational efficiency without drastically increasing model size.

Key takeaway

For AI Scientists and Machine Learning Engineers developing or deploying LLMs, understand that significant performance gains now stem from sophisticated pre-training recipes and architectural optimizations, not just larger models. Focus your efforts on refining data quality, exploring multi-stage training protocols, and integrating efficiency-focused architectural components like GQA or MoE to achieve competitive results, especially with 7B-parameter models.

Key insights

LLM evolution prioritizes data quality, multi-stage pre-training, and architectural efficiency over sheer model size.

Principles

• Data quality and mixing are paramount.
• Multi-stage pre-training enhances model performance.
• Architectural tweaks improve efficiency.

Method

Modern LLM pre-training involves multi-stage procedures: regular pre-training, long-context pre-training, and high-quality data annealing, alongside advanced data filtering and synthesis techniques.

In practice

• Implement RoPE for positional encoding.
• Utilize Grouped Query Attention (GQA).
• Consider Mixture of Experts (MoE) for scale.

Topics

• LLM Architecture Evolution
• Pre-training Pipelines
• Dataset Curation
• Grouped Query Attention
• Mixture-of-Experts

Best for: AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Developing an LLM: Building, Training, Finetuning — LLM development involves building architecture, pre-training on vast datasets, and fine-tuning for specific tasks and behaviors.
• The Architecture Behind Open-Source LLMs — MoE architectures and diverse training strategies drive rapid progress in open-weight LLMs, optimizing cost and performance.
• Data Engineering for the LLM Age — LLM performance fundamentally depends on high-quality, AI-ready data engineering across training, inference, and evaluation.
• How Large Language Models Are Actually Built, Everything I Learned From Stanford’s CS229 LLM… — Data, evaluation, and systems are the true differentiators in LLM quality, not just architecture.
• A Visual Tour of Modern LLM Architectures — The LLM Architecture Gallery offers a visual, comparative resource for understanding diverse LLM designs and their efficiency trade-offs.
• The World Of LLMs Post Scaling Laws — AI model development has shifted from pretraining scale to post-training and verifiable data quality due to data exhaustion.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Sebastian Raschka.