Llama 3 on Your Local Computer | Free GPT-4 Alternative

2024-04-22 · Source: Martin Thissen · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

Meta has released the Llama 3 model, available in 8 billion and 70 billion parameter variants, demonstrating a 10% relative improvement over Llama 2. The 8 billion Llama 3 model can sometimes outperform the Llama 2 70 billion model. Key improvements include an increased context size from 4,000 to 8,000 tokens and training on 15 trillion tokens, seven times more than Llama 2, with four times more code data. Architectural changes are minimal, primarily a new tokenizer with a 128,000-token vocabulary (four times larger than Llama 2's 32,000) and the inclusion of Group Query Attention in the 8 billion variant. These changes enhance token efficiency by up to 15% and maintain inference speed despite increased parameters. The training data, though closed source, includes over 5% high-quality non-English data covering 30+ languages. Meta also developed scaling laws for optimal data mix and utilized advanced instruction fine-tuning techniques like rejection sampling, PPO, and DPO, emphasizing high-quality prompt data.

Key takeaway

For AI Engineers deploying large language models locally, consider Llama 3's 4-bit quantized versions to maximize performance within your GPU's VRAM constraints. The 70 billion parameter model, when 4-bit quantized, requires approximately 37 GB of VRAM, making it accessible on GPUs like the RTX 6000 Ada. Utilize the vLLM library for significantly faster inference speeds compared to standard methods, ensuring a responsive user experience for your applications.

Key insights

Llama 3 achieves significant performance gains through extensive training data, an improved tokenizer, and advanced fine-tuning.

Principles

• Longer training on more data improves model performance.
• Tokenizer vocabulary size impacts model efficiency and performance.
• High-quality instruction tuning data is crucial for model alignment.

Method

Llama 3's improvement pillars include a refined tokenizer, 15 trillion tokens of pre-training data, scaling laws for data mix optimization, and instruction fine-tuning with supervised learning, rejection sampling, PPO, and DPO.

In practice

• Use 4-bit quantized models for larger parameter counts on limited VRAM.
• Prioritize high-quality prompts for instruction fine-tuning.
• Leverage vLLM for faster LLM inference on local GPUs.

Topics

• Llama 3 Model
• LLM Training
• Instruction Fine-tuning
• Local LLM Deployment
• Model Architecture

Best for: Machine Learning Engineer, Deep Learning Engineer, AI Engineer

Related on AIssential

• I Ran a 70B Language Model on My 4GB GPU and It Actually Worked — Here Is What I Found — AirLLM enables running 70B LLMs on 4GB GPUs without quantization, overcoming VRAM limitations.
• Llama 3.2 Vision — Llama 3.2 Vision offers local multimodal AI capabilities via Ollama, supporting diverse image-text tasks.
• llama.cpp: Fast Local LLM Inference, Hardware Choices & Tuning — llama.cpp is the dominant open-source C/C++ framework enabling fast local large language model (LLM) inference, leveraging hardware advancements like NVIDIA's RTX 5090 and Apple's M4 Ultra for privacy…
• What is your current local LLM setup? — The community actively explores diverse local LLM setups, balancing hardware, software, and model choices for varied applications.
• A GPU-Poor’s Guide to Local LLM Inference in 2026 — Local LLM inference on consumer GPUs (4-12 GB VRAM) is now viable for long contexts via MoE models, advanced KV cache quantization, and CPU offloading.
• Run Code Llama locally — Code Llama, based on Llama 2, provides open-source code generation with infilling and large context support.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Martin Thissen.