AI Inference on AMD Ryzen™ AI Max Processor

2026-05-25 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

AMD Ryzen™ AI Max+ processors, featuring AMD Radeon™ 8060S integrated graphics and up to 128 GB of Unified Memory Architecture (UMA), significantly advance local large language model (LLM) inference. This architecture allows the CPU and GPU to share a single memory pool, enabling models with 100 billion or more parameters to run on a single system without dedicated VRAM limitations or cloud costs. The article demonstrates this capability by running Qwen3.5 models (9B, 35B-A3B, and 122B-A10B) on an AMD Ryzen™ AI Max+ 395 system with 64 GB GPU-accessible memory, Ubuntu 24.04 LTS, ROCm 7.2.1, and Ollama 0.20.x. Benchmarks show the 35B-A3B model achieving 42.04 tok/s and the 76 GB 122B-A10B model running at 8.59 tok/s using CPU/GPU mixed loading.

Key takeaway

For AI Engineers evaluating or self-hosting large language models, the AMD Ryzen™ AI Max+ processor offers a compelling local inference solution. Its Unified Memory Architecture allows you to run 100B+ parameter models like Qwen3.5 122B-A10B directly on a single system, eliminating the need for multi-GPU rigs or expensive cloud endpoints. You should consider this hardware for cost-effective, high-capacity LLM development and deployment, leveraging Ollama for streamlined setup and management.

Key insights

AMD Ryzen™ AI Max+ processors enable local 100B+ parameter LLM inference via Unified Memory Architecture, reducing cloud dependency.

Principles

• UMA allows CPU and GPU to share a single memory pool.
• MoE architectures can increase token generation speed.
• GPU-accessible memory is configurable via BIOS settings.

Method

Install Ollama on Ubuntu with AMD ROCm, then pull and run Qwen3.5 models. Verify GPU acceleration with "ollama ps" and monitor memory with "rocm-smi".

In practice

• Use Qwen3.5 35B-A3B for fast interactive chat (~42 tok/s).
• Run Qwen3.5 122B-A10B for large-capacity local workloads (~8.6 tok/s).
• Adjust context length and close memory-intensive apps for large models.

Topics

• LLM Inference
• Unified Memory Architecture
• AMD Ryzen AI Max+
• Ollama
• ROCm
• Qwen3.5
• Local AI

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• AMD's tiny AI PC points to a more local future for model inference — Local AI inference on compact hardware reduces cloud dependency, improves privacy, and lowers latency.
• AMD Instinct™ GPUs MLPerf Inference v6.0 Submission — AMD's MI355X GPU and ROCm stack deliver competitive MLPerf Inference v6.0 results, especially for LLMs and multi-node setups.
• Breaking the Memory Wall in the Age of Inference — Integrating compute directly into SRAM cells via Digital In-Memory Compute (DIMC) effectively addresses AI inference memory bottlenecks.
• A Practical Guide to CPU-Optimized LLM Deployment on Intel® Xeon® 6 Processors on AWS. — Intel Xeon 6 processors with vLLM enable cost-effective, high-throughput LLM inference on CPUs.
• What is LPU? Language Processing Units | The Future of AI Inference — LPUs offer deterministic, low-latency inference for LLMs by optimizing for sequential processing and on-chip memory.
• Dataflow Computing for AI Inference [Kunle Olukotun] - 751 — Reconfigurable Dataflow Architectures optimize AI inference by directly mapping data flow graphs to hardware, maximizing memory bandwidth utilization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.