[P] Bypassing CoreML to natively train a 110M Transformer on the Apple Neural Engine (Orion)

2026-03-05 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, short

Summary

ORION is introduced as the first open end-to-end system enabling direct Apple Neural Engine (ANE) execution and stable multi-step training for large language models, bypassing CoreML's limitations. The project overcame significant challenges, including 11 newly discovered undocumented ANE programming constraints and a "numerical stability ceiling" that caused NaN divergence in previous attempts. Solutions involved a custom compiler with five optimization passes and fixes for stale programs, fp16 overflow, and corrupted weights through deferred compilation, activation clamping, and gradient sanitization. The system achieves 170+ tokens/s for GPT-2 124M inference on an M4 Max and demonstrated stable training of a 110M-parameter Transformer, reducing loss from 12.29 to 6.19 over 1,000 steps with zero NaN occurrences, despite a current bottleneck where each weight update requires a ~4.2s recompilation. This work proves the feasibility of mathematically stable gradient descent directly on Apple's NPU, opening avenues for future advancements like weight patching or incremental compilation.

Key takeaway

ORION is the first open system enabling direct, stable multi-step training of 110M-parameter Transformers on Apple's ANE, bypassing CoreML's opaque abstractions. It achieves a loss drop from 12.29 to 6.19 over 1,000 steps with zero NaN divergence by solving critical fp16 overflow and weight corruption issues via a custom compiler and deferred compilation pipeline. While current recompilation overhead is ~4.2s per step, this validates practical on-device gradient descent on Apple's NPU, opening avenues for future weight patching and incremental compilation.

Topics

• Apple Neural Engine
• On-device Training
• Transformer Models
• Low-level ML
• Custom Compilers

Code references

• mechramc/Orion

Best for: Machine Learning Engineer, NLP Engineer, AI Scientist, AI Engineer, AI Researcher, AI Hardware Engineer

Related on AIssential

• AscendKernelGen: A Systematic Study of LLM-Based Kernel Generation for Neural Processing Units — Domain-specific reasoning and rigorous evaluation are crucial for LLM-based NPU kernel generation.
• Apple M5 Chip Hardware Explained — The Apple M5 chip integrates distributed neural accelerators directly into GPU cores for enhanced AI processing.
• TinyML Series: Article 01 — TinyML enables on-device machine learning inference for resource-constrained edge devices, prioritizing privacy, low latency, and offline functionality.
• ORBIS: Output-Guided Token Reduction with Distribution-Aware Matching for Video Diffusion Acceleration — SW-HW co-design with output-guided, distribution-aware token reduction significantly accelerates video Diffusion Transformers by exploiting temporal consistency.
• Apple Machine Learning Research at ICLR 2026 — Apple's ICLR 2026 research focuses on scaling efficient AI models and unifying multimodal capabilities.
• Accelerating on-device AI: A look at Arm and Google AI Edge optimization — Arm SME2 and Google AI Edge enable efficient, high-quality on-device generative AI inference on CPUs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.