Ollama now supports AMD graphics cards

2024-03-13 · Source: Ollama Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Fundamental Awareness, quick

Summary

Ollama has introduced preview support for AMD graphics cards on both Windows and Linux, enabling acceleration for all Ollama features. This update, released on March 14, 2024, significantly expands the hardware compatibility for running large language models locally. Supported AMD Radeon RX cards include the 7900 XTX, 7900 XT, and 6950 XT, among others. AMD Radeon PRO series like the W7900 and W6800X are also supported, alongside AMD Instinct accelerators such as the MI300X and MI250X. Ollama plans to extend support to additional AMD graphics cards in future updates, broadening access for users with AMD hardware.

Key takeaway

For developers and researchers using AMD hardware for local AI inference, you can now leverage Ollama's full feature set with GPU acceleration on Windows and Linux. This update removes a significant barrier for those without NVIDIA GPUs, allowing you to experiment with and deploy large language models more efficiently. Download the latest Ollama version to begin utilizing your AMD Radeon RX, Radeon PRO, or Instinct cards.

Key insights

Ollama now supports AMD GPUs on Windows and Linux, expanding local LLM accessibility.

Principles

• Broaden hardware compatibility
• Enable local model acceleration

Method

Download and install Ollama for Linux or Windows to utilize AMD GPU acceleration for local large language models.

In practice

• Run LLMs on AMD Radeon RX GPUs
• Accelerate models with AMD Instinct cards

Topics

• Ollama
• AMD GPUs
• AI Acceleration
• Local LLMs
• Windows/Linux Support

Code references

• ollama/ollama

Best for: NLP Engineer, AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• Windows preview — Ollama's Windows preview enables native, accelerated local LLM execution with full model library and OpenAI-compatible API access.
• Improved performance and model support with GGUF — Ollama 0.30 significantly boosts performance and broadens model/hardware compatibility via GGUF and llama.cpp integration.
• vLLM-ATOM: Unlocking Native AMD Performance in the vLLM Ecosystem — vLLM-ATOM integrates AMD's ATOM engine as a plugin to optimize vLLM inference on Instinct GPUs.
• What is your current local LLM setup? — The community actively explores diverse local LLM setups, balancing hardware, software, and model choices for varied applications.
• Ollama: Unleash the Power of Local LLMs on Your Machine — Ollama enables private, cost-effective, and low-latency local LLM inference and customization via a Python library.
• AI Inference on AMD Ryzen™ AI Max Processor — AMD Ryzen™ AI Max+ processors enable local 100B+ parameter LLM inference via Unified Memory Architecture, reducing cloud dependency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Ollama Blog.