Hosting Qwen on Blackwell - Perplexity

2026-05-12 · Source: perplexity.ai via Google News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Expert, long

Summary

Perplexity has successfully deployed and is serving post-trained Qwen3 235B models on NVIDIA GB200 NVL72 racks, leveraging Blackwell GPUs and 72-way NVLink interconnects to handle significant traffic at reduced cost. The deployment utilizes 18 nodes, each with 2 NVIDIA Grace CPUs and 4 Blackwell GPUs, totaling 72 GPUs interconnected by NVLink and 18 NVLink Switch ASICs, providing 1800 GB/s bandwidth. Perplexity adapted its in-house inference engine, TransferEngine, to disaggregate prefill and decode operations, using InfiniBand for prefiller-to-decoder communication and NVLink for intra-node communication. This setup allows for distinct parallelism strategies: tensor parallelism (TP=4, EP=4) for compute-intensive prefill and data/expert parallelism (EP=16) for memory-bound decode, with Blackwell showing kernel-by-kernel improvements over Hopper GPUs. The deployment uses MXFP8 quantization for Qwen3 235B, achieving improved performance with similar accuracy to block scaling.

Key takeaway

For AI Architects and ML Engineers evaluating next-generation inference infrastructure, the successful deployment of Qwen3 235B on NVIDIA Blackwell GB200 NVL72 racks demonstrates significant throughput and cost efficiency gains. You should consider Blackwell's large NVLink domains and hardware-accelerated features like SHARP and MXFP8 quantization for scaling large Mixture-of-Experts (MoE) models, particularly when disaggregating prefill and decode stages to optimize resource utilization and reduce latency.

Key insights

Blackwell GPUs with NVLink enable efficient, cost-effective deployment of large MoE models like Qwen3 235B.

Principles

• Disaggregate prefill and decode for optimal parallelism.
• Utilize NVLink Switches for efficient all-reduce operations.
• Microscaling quantization improves Blackwell throughput.

Method

Deploy MoE models on GB200 NVL72 by disaggregating prefill (tensor parallel, TP=4, EP=4) and decode (data/expert parallel, EP=16), using InfiniBand for inter-stage and NVLink/SHARP for intra-stage communication, and MXFP8 quantization.

In practice

• Consider GB200 NVL72 for large MoE model inference.
• Implement disaggregated prefill/decode for LLM serving.
• Explore MXFP8 quantization for Blackwell deployments.

Topics

• NVIDIA Blackwell GPUs
• Qwen Models
• Mixture-of-Experts
• LLM Inference Optimization
• NVLink Interconnect

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Qwen3.5 9B MoQ: Inside a Strong 3.6-bit GGUF — Advanced quantization and MoE architectures significantly boost LLM efficiency and performance on diverse hardware.
• FlashAttention-4 gives the NVIDIA Blackwell platform its most optimized attention kernel yet — FlashAttention-4 optimizes transformer attention for NVIDIA Blackwell GPUs, significantly boosting performance and hardware utilization.
• NVIDIA Blackwell Sets STAC-AI Record for LLM Inference in Finance — NVIDIA Blackwell architecture significantly improves LLM inference performance for financial RAG workloads, outperforming Hopper by up to 3.2x.
• Develop Native Multimodal Agents with Qwen3.5 VLM Using NVIDIA GPU-Accelerated Endpoints — Qwen3.5 is a ~400B parameter open-source VLM with a hybrid MoE and Gated Delta Network architecture, excelling in UI navigation.
• not much happened today — AI progress is driven by aggressive model scaling, local inference optimization, and maturing agentic workflows.
• Qwen3.5-397B-A17B: the smallest Open-Opus class, very efficient model — Qwen3.5 offers multimodal, long-context capabilities with efficient sparse MoE architecture, enabling local deployment despite its large scale.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by perplexity.ai via Google News.