Introducing Disaggregated Inference on AWS powered by llm-d

2026-03-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, long

Summary

AWS has partnered with the llm-d team to integrate disaggregated inference capabilities into AWS services, enhancing performance and cost efficiency for large-scale LLM inference workloads. This collaboration introduces a new container, `ghcr.io/llm-d/llm-d-aws`, which includes AWS-specific libraries like Elastic Fabric Adapter (EFA) and libfabric, alongside NIXL integration for multi-node disaggregated inference and expert parallelism. The llm-d framework, built on vLLM, is Kubernetes-native and optimizes LLM serving by separating prefill and decode phases, enabling intelligent request scheduling, and supporting tiered prefix caching. Benchmarking shows that llm-d's prefill/decode disaggregation can increase tokens per second by up to 70% compared to standard vLLM deployments under specific load conditions, particularly for workloads with long input and output sequences.

Key takeaway

For MLOps Engineers deploying large language models at scale on AWS, adopting llm-d's disaggregated inference architecture on Amazon SageMaker HyperPod or EKS can significantly improve throughput and reduce costs. You should explore configuring prefill/decode disaggregation and intelligent scheduling to optimize resource utilization and achieve up to 70% higher tokens per second for agentic or long-sequence workloads.

Key insights

Disaggregated LLM inference on AWS significantly boosts performance and resource utilization for large-scale AI deployments.

Principles

• Separate compute-bound prefill from memory-bound decode.
• Utilize cache-aware routing for distributed LLM inference.
• Offload KV cache entries beyond GPU memory limits.

Method

llm-d orchestrates distributed LLM serving by disaggregating prefill and decode phases, employing intelligent, cache-aware scheduling, and leveraging high-speed interconnects like EFA via NIXL for efficient KV cache transfers.

In practice

• Deploy llm-d on Amazon SageMaker HyperPod or EKS.
• Configure EFA interfaces for high-bandwidth GPU communication.
• Tune prefill/decode ratios for specific workload characteristics.

Topics

• LLM Inference Optimization
• Disaggregated Serving
• Kubernetes
• vLLM
• Elastic Fabric Adapter

Code references

• vllm-project/vllm
• llm-d/llm-d

Best for: MLOps Engineer, AI Engineer, Machine Learning Engineer

Related on AIssential

• A Deep Dive Into Disaggregation in LLMs — Disaggregated LLM inference optimizes performance and cost by separating prefill, decode, and routing stages.
• Best practices to run inference on Amazon SageMaker HyperPod — SageMaker HyperPod optimizes generative AI inference with dynamic scaling, intelligent caching, and efficient GPU utilization.
• AWS and Cerebras partner to advance AI inference performance in the cloud — Inference disaggregation optimizes generative AI by dedicating specialized hardware to prefill and decode stages.
• Accelerate custom LLM deployment: Fine-tune with Oumi and deploy to Amazon Bedrock — Oumi and AWS services streamline LLM fine-tuning and deployment from experimentation to managed production inference.
• Lodestar: An Online-Learning LLM Inference Router — Lodestar uses online learning to dynamically route LLM inference requests, significantly reducing latency by adapting to real-time cluster conditions.
• Running AI on mixed hardware for speed and affordability — llm-d optimizes LLM inference on mixed GPU clusters by intelligently routing requests based on KV cache state, significantly boosting throughput and reducing costs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.