The Hummingbird That Carries a Giant: Inside Colibrì, the 744B-Parameter Model Running on a Laptop…

2026-07-12 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, medium

Summary

Colibrì, a pure-C project, enables running the 744-billion-parameter GLM-5.2 Mixture-of-Experts model on consumer hardware with approximately 25GB of RAM, without requiring a GPU, Python runtime, or BLAS. It achieves this by treating the model's 21,504 routed experts as a streaming, cacheable resource on disk, while keeping a dense backbone resident in RAM, quantized to int4, taking up under 10GB. Key techniques include a per-layer LRU cache, an optional pinned "hot store," and leveraging the OS page cache. The implementation is token-exact, features MLA attention with LoRA, a 57x compressed KV-cache, and a native multi-token-prediction head for speculative decoding, achieving 2.00 tokens per pass. While cold decoding is slow (0.05-0.1 tokens/second), estimates suggest 5-15 tokens/second on better workstations. It also includes an offline FP8-to-int4 converter.

Key takeaway

For AI Engineers and Systems Engineers optimizing large model inference on memory-constrained hardware, Colibrì demonstrates that treating Mixture-of-Experts model components as streamable, cacheable resources can drastically reduce memory footprints. You should explore disk-based caching strategies and specialized inference engines to push the boundaries of what consumer-grade hardware can achieve with frontier-scale models, especially when full model loading is impractical.

Key insights

Model size and memory requirements are not identical constraints for Mixture-of-Experts models.

Principles

• MoE expert weights can be treated as streamable, cacheable resources.
• Optimize disk I/O to make on-demand expert loading feel like memory access.
• Apply data-engineering principles like working sets to model serving problems.

Method

Colibrì keeps a dense model backbone in RAM and streams 21,504 routed experts from disk on demand, utilizing LRU, hot store, and OS page caches.

In practice

• Implement LRU caching and hot-set pinning for disk-resident model components.
• Utilize compressed KV-caches for significant per-token storage reduction.
• Develop offline converters for large checkpoints to manage storage logistics.

Topics

• Mixture-of-Experts
• Large Language Models
• Memory-constrained Inference
• Disk I/O Optimization
• C Programming
• GLM-5.2
• Quantization

Best for: NLP Engineer, Entrepreneur, AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• The Brain Trick Behind the World’s Best AI Models — Mixture of Experts decouples model capacity from inference cost by activating only a subset of specialized networks per token.
• KV Cache - Explained — Caching Key and Value vectors eliminates quadratic attention recomputation, making LLM inference linear per step.
• Streaming experts — Streaming experts allows large MoE models to run on RAM-limited hardware by loading weights from SSD on demand.
• Less is MoE: Trimming Experts in Domain-Specialist Language Models — Fisher importance-based pruning of FFN intermediate dimensions effectively compresses MoE models, preserving capability and improving inference.
• HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference — HCRMap dynamically manages hot expert replicas to mitigate resource bottlenecks in MoE LLM inference on 3.5D chiplets.
• Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — On bandwidth-bound edge hardware, MoE inference cost tracks total parameters, not active ones, negating expected efficiency gains.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.