Native-speed vLLM transformers modeling backend

2026-07-08 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, short

Summary

The "transformers" vLLM backend, updated on July 8, 2026, now achieves native-speed inference, matching or exceeding custom vLLM implementations for many Large Language Model architectures. This significant enhancement allows model authors to automatically utilize their existing "transformers" implementations for ultra-fast vLLM inference without requiring any code porting. Benchmarks against Qwen3 models—including a 4B dense model on a single GPU, a 32B dense model with tensor parallelism, and a 235B-parameter FP8 Mixture-of-Experts model on an 8×H100 node—demonstrated that the "transformers" backend consistently meets or surpasses native throughput. The improvement stems from dynamically applying inference-specific layer fusions at runtime, leveraging "torch.fx" for static graph analysis and "ast" for source code manipulation to optimize operations into highly efficient vLLM kernels, supporting features like expert and tensor parallelism.

Key takeaway

For MLOps Engineers deploying LLMs, this "transformers" vLLM backend update simplifies achieving native inference speeds. You can now deploy "transformers" models directly into vLLM with "--model-impl transformers", eliminating the need for custom vLLM ports and maintaining performance parity. This streamlines your deployment workflow, allowing you to use the same model code for both training and high-performance inference. Ensure your "vllm" pip package is upgraded to leverage these optimizations.

Key insights

"transformers" models now achieve native vLLM inference speeds through dynamic runtime optimizations, eliminating manual porting.

Principles

• "transformers" is the reference modeling library.
• Optimized inference requires specific layer fusions.
• Same model code for training and inference.

Method

The "transformers" vLLM backend uses "torch.fx" for static graph analysis to identify optimizable patterns. It then employs "ast" to manipulate source code, rewriting operations for fused vLLM kernels, including for expert and tensor parallelism.

In practice

• Use "--model-impl transformers" flag.
• Apply parallelism options like "--tensor-parallel-size".
• Upgrade "vllm" pip package.

Topics

• vLLM
• Hugging Face Transformers
• LLM Inference Optimization
• Model Deployment
• torch.fx
• Parallel Computing

Code references

• huggingface/blog

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Optimizing LLM latency — MLC, CTranslate2, and vLLM offer significant latency improvements for LLM inference, with varying trade-offs.
• I Deployed Local LLMs in Production for a Year. Part 1: The Mental Model — Production LLM deployment requires understanding hidden queues, KV cache memory, and multi-phase model loading beyond basic tutorials.
• How to Cache vLLM Model in FastAPI for Faster Inference — Caching VLM/LLM models in FastAPI's global scope significantly reduces inference latency by avoiding repeated initialization.
• A Deep Dive Into Disaggregation in LLMs — Disaggregated LLM inference optimizes performance and cost by separating prefill, decode, and routing stages.
• Transformers.js v4: State-of-the-art machine learning for the web — Transformers.js v4 significantly boosts performance and model scale via a new C++ WebGPU backend and expanded architecture support.
• Accelerating Multimodal Inference in vLLM: The One-Line Optimization for Large Multimodal Models — Batch-level Data Parallelism in vLLM significantly boosts multimodal inference by reducing vision encoder communication overhead.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.