How to serve Kimi-K2-Instruct on Lambda with vLLM

2025-12-22 · Source: The Lambda Deep Learning Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, short

Summary

This article details the deployment of Moonshot AI's Kimi-K2-Instruct, a one-trillion-parameter Mixture-of-Experts (MoE) language model, on Lambda's 8x NVIDIA Blackwell GPU instances using vLLM for efficient multi-GPU inference. Kimi-K2-Instruct, licensed under MIT, features a 128K context window and requires 959GB of disk space and 1,347GB of idle vRAM, making single-GPU operation impractical. The deployment process involves spinning up an 8x NVIDIA Blackwell GPU instance, configuring a vLLM server with specific parameters like `--tensor-parallel-size 8`, and then benchmarking the server using `vllm bench serve`. Benchmark results show an output generation throughput of 219.644 ± 0.497 tokens per second and a total throughput of 332.358 ± 0.810 tokens per second, with a mean time to first token of 148.986 ± 6.619 ms.

Key takeaway

For MLOps Engineers deploying large language models that exceed single-GPU memory, you should consider multi-GPU serving solutions like vLLM on cloud instances such as Lambda's 8x Blackwell GPUs. This approach enables efficient inference for models like Kimi-K2-Instruct, which demands over a terabyte of VRAM. Ensure you benchmark your deployment to validate performance metrics like time-to-first-token and overall throughput before production.

Key insights

Deploying large MoE models like Kimi-K2-Instruct requires multi-GPU inference solutions such as vLLM on specialized hardware.

Principles

• Large models necessitate multi-GPU serving.
• Benchmarking is crucial for production efficiency.

Method

Deploy Kimi-K2-Instruct by launching an 8x Blackwell GPU instance on Lambda, starting a vLLM server with `tensor-parallel-size 8`, and then benchmarking using `vllm bench serve` with sleep mode enabled for accurate measurements.

In practice

• Use `tensor-parallel-size 8` for 8 GPUs.
• Employ `vllm bench serve` for throughput metrics.
• Activate sleep mode to reset server state.

Topics

• Kimi-K2-Instruct
• vLLM
• Multi-GPU Inference
• Mixture-of-Experts
• LLM Deployment

Best for: Machine Learning Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• A GPU-Poor’s Guide to Local LLM Inference in 2026 — Local LLM inference on consumer GPUs (4-12 GB VRAM) is now viable for long contexts via MoE models, advanced KV cache quantization, and CPU offloading.
• Kimi K2 Thinking: what 200+ tool calls mean for production — Kimi K2 Thinking is an open-source MoE reasoning model capable of 200-300 sequential tool calls.
• Build with Kimi K2.5 Multimodal VLM Using NVIDIA GPU-Accelerated Endpoints — Kimi K2.5 is a multimodal VLM optimized for agentic AI and diverse tasks, built on Megatron-LM with a specialized vision tower.
• How to Deploy Your LLM in the Cloud — Self-hosting LLMs offers greater control and cost optimization compared to serverless solutions for production deployments.
• lyogavin / airllm — AirLLM enables running large language models on low-VRAM GPUs by optimizing inference memory.
• [AINews] not much happened today — AI inference efficiency, open model releases, and production-ready agentic workflows are rapidly advancing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Lambda Deep Learning Blog.