Fast transformer inference with Metal Performance Shaders

2022-11-24 · Source: Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, quick

Summary

Thinc PyTorch layers now feature integrated support for Metal Performance Shaders (MPS), a significant update that enables GPU-accelerated inference for spaCy transformer-based pipelines on Apple Silicon Macs. This enhancement directly addresses the performance needs of users within the Apple ecosystem, allowing them to utilize their integrated GPUs for demanding natural language processing tasks. The new support dramatically improves inference speed, with reported gains of up to 4.7 times compared to previous configurations. This development streamlines the execution of complex transformer models, making local development and deployment more efficient and responsive for professionals working with spaCy on Apple's custom silicon architecture.

Key takeaway

For NLP Engineers developing or deploying spaCy transformer pipelines on Apple Silicon Macs, you should update your Thinc PyTorch layers to leverage Metal Performance Shaders. This integration offers a substantial performance uplift, potentially accelerating your inference tasks by up to 4.7 times. Utilizing this feature will significantly reduce processing times for local development and testing, making your workflow more efficient and responsive. Ensure your environment is configured to take advantage of the native GPU support.

Key insights

Metal Performance Shaders in Thinc PyTorch layers accelerate spaCy transformer inference on Apple Silicon Macs by up to 4.7x.

Principles

• Hardware-specific optimization boosts performance.
• GPU acceleration is critical for transformer inference.

In practice

• Run spaCy transformers on Apple Silicon GPUs.
• Achieve up to 4.7x faster inference locally.

Topics

• Metal Performance Shaders
• Thinc PyTorch
• spaCy
• Transformer Models
• Apple Silicon
• GPU Acceleration
• Inference Optimization

Best for: Machine Learning Engineer, NLP Engineer, AI Engineer

Related on AIssential

• Sparser, Faster, Lighter Transformer Language Models — Reshaping LLM sparsity to fit GPU architecture significantly improves inference and training speed.
• Vibe coding is about to get so fast — GPT 5.3 Spark leverages Cerebras wafer-scale chips for unprecedented coding inference speed.
• Customizing Kernels with hipBLASLt TensileLite GEMM Tuning - Advanced User Guide — TensileLite Tuning generates custom GEMM kernels for AMD GPUs, significantly outperforming existing solutions.
• [P] Bypassing CoreML to natively train a 110M Transformer on the Apple Neural Engine (Orion) — ORION is introduced as the first open end-to-end system enabling direct Apple Neural Engine (ANE) execution and stable multi-step training for large language models, bypassing CoreML's limitations.
• Burning a Transformer into Silicon: The Case for GPU-Free AI Inference — Directly burning transformer logic into FPGAs offers ultra-efficient, GPU-free AI inference.
• ORBIS: Output-Guided Token Reduction with Distribution-Aware Matching for Video Diffusion Acceleration — SW-HW co-design with output-guided, distribution-aware token reduction significantly accelerates video Diffusion Transformers by exploiting temporal consistency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Explosion · Developer tools and consulting for AI, Machine Learning and NLP - Explosion.ai.