BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

2025-02-17 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Expert, long

Summary

BaseRT, a native Metal inference runtime for large language models on Apple Silicon, has been extended to exploit the M5 generation's redesigned GPU architecture. The M5 introduces dedicated Neural Accelerators, on-die matrix units exposed via the Metal 4 tensor API, which BaseRT now utilizes through a family of hand-written Metal 4 tensor-core kernels. These kernels route compute-bound matrix multiplications, including dense and mixture-of-experts (MoE) GEMM and flash-attention prefill, through the M5 Neural Accelerators. On an Apple M5 Pro, BaseRT delivers up to \$6.4\times$ higher prompt-processing throughput than llama.cpp and \$3.9\times$ higher than MLX across fifteen model configurations, spanning sub-1B to 35B parameters from Qwen3, Qwen3.5/3.6, Llama 3.2, and Gemma 4 families. It also maintains a decode lead of up to \$1.75\times$ over llama.cpp and \$1.33\times$ over MLX, establishing a new performance ceiling for on-device LLM inference.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing LLM inference on Apple Silicon, you should prioritize BaseRT for M5-based deployments. Its specialized Metal 4 tensor-core kernels significantly boost prompt-processing throughput, up to \$6.4\times$ over llama.cpp, especially for compute-intensive tasks like MoE models. While decode gains are modest due to memory bandwidth limits, BaseRT offers a clear advantage for reducing time-to-first-token in long-prompt and agentic workloads. Consider integrating BaseRT to maximize M5 hardware utilization.

Key insights

BaseRT utilizes Apple M5 Neural Accelerators via Metal 4 tensor API for superior LLM inference performance.

Principles

• Dedicated matrix units boost compute-bound tasks.
• Workload-aware dispatch optimizes performance.
• Memory-bound tasks gain less from tensor cores.

Method

BaseRT extends its framework-free design with hand-written Metal 4 tensor-core kernels for compute-bound operations (GEMM, MoE, prefill attention) and uses existing kernels for memory-bound decode.

In practice

• Implement Metal 4 tensor-core kernels for M5 GPUs.
• Apply tensor cores to prefill and MoE inference.
• Maintain specialized kernels for memory-bound decode.

Topics

• LLM Inference
• Apple M5 Neural Accelerators
• Metal 4 Tensor API
• BaseRT Runtime
• Prompt Processing
• Mixture-of-Experts Models

Code references

• basecompute/baseRT
• ggml-org/llama.cpp
• ml-explore/mlx

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators — BaseRT leverages Apple M5 Neural Accelerators and Metal 4 tensor API for significantly faster LLM inference on Apple Silicon.
• How We Cut LLM Latency 70% With TensorRT in Production — Effective enterprise AI deployment requires rigorous cost management, performance optimization, and a strategic approach to proving ROI.
• Building a monokernel for LLM inference on AMD MI300X - up to 3,300 output tokens/s per request [P] — Optimizing LLM inference on AMD MI300X by leveraging hardware die topology with a single GPU-resident monokernel.
• MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling — MARS uses margin-adversarial stopping to reduce LLM test-time scaling costs by predicting trace stability and stopping early.
• Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results — Amplitude Gating (AG) improves LLM structured outputs by modulating FFN activation magnitudes during inference without retraining.
• vLLM vs Triton vs TGI: Choosing the Right LLM Serving Framework — Efficient LLM inference requires optimizing KV cache management and batching for high throughput and low latency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.