lyogavin / airllm

2023-06-12 · Source: Github Trending: All languages · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

AirLLM is an open-source tool designed to significantly optimize large language model inference memory usage, enabling models like 70B LLMs to run on a single 4GB GPU card without initial quantization, distillation, or pruning. It also supports running the 405B Llama3.1 model on 8GB VRAM. The tool now includes block-wise quantization-based model compression, which can accelerate inference speed by up to 3x with minimal accuracy loss, primarily by reducing model loading size. AirLLM supports a wide range of models, including Llama3, Qwen2.5, ChatGLM, Baichuan, Mistral, and InternLM, and offers CPU inference and MacOS support for 70B models. Key features include an "AutoModel" for automatic type detection and prefetching for a 10% speed improvement.

Key takeaway

For MLOps Engineers deploying large language models on resource-constrained hardware, AirLLM provides a critical solution. You can now run 70B LLMs on a single 4GB GPU or 405B Llama3.1 on 8GB VRAM, drastically reducing VRAM requirements without complex quantization. Consider integrating AirLLM to lower infrastructure costs and expand deployment possibilities for powerful models, especially for local or edge inference, while potentially achieving 3x speed improvements with its compression features.

Key insights

AirLLM enables running large language models on low-VRAM GPUs by optimizing inference memory.

Principles

• Memory optimization can bypass full quantization for large LLMs.
• Block-wise weight quantization boosts inference speed up to 3x.

Method

Install "airllm", initialize "AutoModel.from_pretrained()" with a model ID, then call "model.generate()". Enable compression via "compression='4bit'" or "'8bit'".

In practice

• Run 70B LLMs on a single 4GB GPU.
• Deploy 405B Llama3.1 on 8GB VRAM.
• Achieve 3x inference speed-up with compression.

Topics

• LLM Inference Optimization
• Low-VRAM Deployment
• Block-wise Quantization
• Llama3.1
• GPU Memory Management
• AirLLM

Code references

• lyogavin/airllm
• LianjiaTech/BELLE
• sponsors/lyogavin
• TimDettmers/bitsandbytes
• ml-explore/mlx

Best for: NLP Engineer, Entrepreneur, Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• I Ran a 70B Language Model on My 4GB GPU and It Actually Worked — Here Is What I Found — AirLLM enables running 70B LLMs on 4GB GPUs without quantization, overcoming VRAM limitations.
• Inference Optimization — How to Make LLMs Faster and Cheaper in Production — Optimizing LLM inference in production requires a stack of techniques to balance latency, throughput, and cost.
• How to serve Kimi-K2-Instruct on Lambda with vLLM — Deploying large MoE models like Kimi-K2-Instruct requires multi-GPU inference solutions such as vLLM on specialized hardware.
• Llama 3 on Your Local Computer | Free GPT-4 Alternative — Llama 3 achieves significant performance gains through extensive training data, an improved tokenizer, and advanced fine-tuning.
• Qwen3.5 9B MoQ: Inside a Strong 3.6-bit GGUF — Advanced quantization and MoE architectures significantly boost LLM efficiency and performance on diverse hardware.
• 2026 Predictions: Much Faster Inference, Pre-Training with RL, and FP4 Everywhere — DeepSeek's MoE and RLVR advancements, alongside FP4 quantization, defined 2025 LLM progress, while Llama 4 disappointed.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Github Trending: All languages.