Serving DeepSeek-V4: why million-token context is an inference systems problem

2026-05-11 · Source: Together AI | The AI Native Cloud - Together.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

DeepSeek-V4 introduces a 1M-token context window through a novel hybrid attention architecture, transforming long-context capabilities into an inference systems challenge. Its design compresses context before Key-Value (KV) storage, mixing Compressed Sparse Attention (CSA) (stride 4, 8-token neighborhoods) and Heavily Compressed Attention (HCA) (stride 128, 8K entries for 1M tokens) with Sliding Window Attention (SWA) (128 tokens). This approach significantly reduces KV cache pressure, improving concurrency and throughput. However, realizing these gains requires sophisticated inference engine management of diverse cache layouts, complex prefix reuse policies, and workload-specific endpoint profiles. Together's early work on NVIDIA HGX B200 demonstrated increasing KV-cache capacity from 1.2M to 3.7M tokens by optimizing SWA state, highlighting the critical role of serving policy.

Key takeaway

For MLOps Engineers deploying DeepSeek-V4, recognize that its 1M-token context efficiency is a system property, not an out-of-the-box feature. You must actively manage diverse KV cache layouts, optimize prefix reuse policies (e.g., SWA recompute), and configure workload-specific endpoint profiles. Benchmark your specific context-length regime and traffic shapes to unlock V4's architectural savings, ensuring lower latency and higher concurrency for long-context applications like coding agents.

Key insights

DeepSeek-V4's 1M-token context is an inference systems problem, demanding specialized cache management for efficiency.

Principles

• Long-context efficiency is a system property.
• KV cache compression improves concurrency and throughput.
• Workload-specific serving profiles optimize performance.

Method

DeepSeek-V4 employs a hybrid attention design (CSA, HCA, SWA) to compress context before KV storage, mixing compressed and local attention paths, and adapting prefix reuse strategies.

In practice

• Benchmark V4 in your actual context-length regime.
• Test SWA full-store vs. recompute-on-hit for prefixes.
• Configure endpoint profiles for specific traffic shapes.

Topics

• DeepSeek-V4
• Inference Systems
• KV Cache Optimization
• Hybrid Attention
• Long Context LLMs
• NVIDIA Blackwell

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• The Sequence AI of the Week #851: DeepSeek-V4 and the Architecture of Million-Token Intelligence — Effective long-context reasoning requires a holistic systems approach beyond just scaling Transformer models.
• KV Cache Explained Like You’re an LLM Engineer — KV cache makes LLM inference viable by storing past Key/Value tensors, avoiding redundant attention recomputation.
• DeepSeek launches V4 model with one million token context — DeepSeek-V4 offers enhanced AI capabilities, including a one-million-token context, optimized for domestic chips.
• DeepSeek-V4 Review: Why Million-Token Context Needs Efficient Attention, Not Just Larger Windows — Efficient attention mechanisms and specialized training are key to practical 1M-token context LLMs.
• DeepSeek V4 AI Beats Billion Dollar Systems…For Free — DeepSeek 4 achieves massive context windows and efficiency through multi-layered KV-cache compression.
• DeepSeek-V4: a million-token context that agents can actually use — DeepSeek V4 optimizes large language models for agentic workflows through architectural and training innovations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Together AI | The AI Native Cloud - Together.ai.