BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

2026-07-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

BaseRT, a native Metal inference runtime for large language models on Apple Silicon, significantly boosts inference throughput on Apple M5 Neural Accelerators. The M5 generation features a redesigned GPU architecture with dedicated Neural Accelerators, exposed via the Metal 4 tensor API. BaseRT leverages a framework-free design and hand-written Metal 4 tensor-core kernels for compute-bound matrix multiplications, including dense, mixture-of-experts GEMM, and flash-attention prefill. On an Apple M5 Pro, BaseRT delivers up to 6.4x higher prompt-processing throughput than llama.cpp and 3.9x higher than MLX across fifteen model configurations (Qwen3, Qwen3.5/3.6, Llama 3.2, Gemma 4 families, sub-1B to 35B parameters). It also maintains a decode lead of up to 1.75x over llama.cpp and 1.33x over MLX, establishing a new performance ceiling for on-device LLM inference. BaseRT is publicly available on GitHub.

Key takeaway

For Machine Learning Engineers optimizing LLM inference on Apple Silicon, BaseRT offers substantial performance gains. Its specialized Metal 4 tensor-core kernels exploit the M5 Neural Accelerators, delivering up to 6.4x higher prompt-processing throughput than llama.cpp. You should consider integrating BaseRT into your M5-based LLM deployments to achieve best-in-class on-device performance, especially for compute-intensive tasks like prompt processing.

Key insights

BaseRT leverages Apple M5 Neural Accelerators and Metal 4 tensor API for significantly faster LLM inference on Apple Silicon.

Principles

• Dedicated Neural Accelerators boost LLM inference.
• Framework-free design enables specialized kernel optimization.
• Tensor cores are critical for prompt processing performance.

Method

BaseRT uses hand-written Metal 4 tensor-core kernels for compute-bound matrix multiplications (dense, MoE GEMM, flash-attention prefill) and existing specialized kernels for memory-bound decode.

In practice

• Use BaseRT for LLM inference on Apple M5.
• Optimize matrix multiplication with tensor cores.
• Separate compute-bound from memory-bound paths.

Topics

• LLM Inference
• Apple Silicon
• Neural Accelerators
• Metal 4 API
• BaseRT
• Performance Optimization
• Mixture-of-Experts

Code references

• basecompute/baseRT

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators — BaseRT utilizes Apple M5 Neural Accelerators via Metal 4 tensor API for superior LLM inference performance.
• Hot French startup ZML releases free product to speed inference across lots of AI chips — ZML/LLMD offers cross-platform LLM inference optimization to break hardware silos and reduce AI operational costs.
• Building a monokernel for LLM inference on AMD MI300X - up to 3,300 output tokens/s per request [P] — Optimizing LLM inference on AMD MI300X by leveraging hardware die topology with a single GPU-resident monokernel.
• Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results — Amplitude Gating (AG) improves LLM structured outputs by modulating FFN activation magnitudes during inference without retraining.
• MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling — MARS uses margin-adversarial stopping to reduce LLM test-time scaling costs by predicting trace stability and stopping early.
• vLLM vs Triton vs TGI: Choosing the Right LLM Serving Framework — Efficient LLM inference requires optimizing KV cache management and batching for high throughput and low latency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.