Running Granite 4 Language Models with Ollama

2026-01-05 · Source: Niklas Heidloff · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

IBM has released the Granite 4 family of open-source language models, which significantly reduce memory requirements for AI inference by up to 70% compared to traditional Transformer-based models. This efficiency is achieved through a hybrid architecture combining Transformer and Mamba layers, allowing for linear scaling of context windows rather than quadratic. The Granite 4 models come in various sizes, including 32B (9B activated), 7B (1B activated), 3B, 1.5B, and 350M parameters, catering to diverse applications from enterprise RAG to edge computing. These models support features like tool calling, RAG, FIM, and structured output, and can be run locally via Ollama, with quantized Q4 versions suitable for most modern notebooks. The smallest models can even operate in browsers using Transformers.js, WebGPU, and ONNX.

Key takeaway

For AI Engineers and Machine Learning Engineers focused on optimizing inference costs and deploying LLMs on resource-constrained hardware, the IBM Granite 4 models offer a compelling solution. Their hybrid Transformer/Mamba architecture provides up to a 70% reduction in memory usage, enabling local execution via Ollama or even in-browser deployment. You should explore these models for applications requiring large context windows, low latency, or edge deployment to significantly lower operational costs and expand accessibility.

Key insights

IBM's Granite 4 models use a hybrid Transformer/Mamba architecture to drastically cut AI inference memory needs.

Principles

• Hybrid architectures improve LLM efficiency.
• Mamba layers enable linear context scaling.
• Smaller models can outperform larger predecessors.

Method

Run Granite 4 models locally using Ollama, specifying quantization (e.g., Q4_K_M or Q8). For browser-based use, leverage Transformers.js, WebGPU, and ONNX.

In practice

• Deploy Granite 4 for low-latency local applications.
• Utilize Granite 4 for RAG, agents, and function calling.
• Experiment with browser-based AI code completion.

Topics

• IBM Granite 4.0
• Hybrid Mamba Architecture
• Local LLM Inference
• Memory Efficiency
• Edge AI

Best for: AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• IBM Granite 3.0 models — IBM's Granite 3.0 models, including dense and MoE variants, are now available via Ollama under Apache 2.0.
• Mamba-3 — Mamba-3 optimizes SSMs for inference efficiency through enhanced recurrence, complex states, and MIMO, surpassing prior models in speed and accuracy.
• Olmo Hybrid and future LLM architectures — Non-attention architectures are challenging Transformers by offering better scaling and efficiency for long-context LLMs.
• Ollama: Unleash the Power of Local LLMs on Your Machine — Ollama enables private, cost-effective, and low-latency local LLM inference and customization via a Python library.
• Granite 4.1, IBM Bob & building a quantum ecosystem — Enterprise AI is shifting from monolithic models to composable, specialized systems for cost-effective, sustainable operations.
• The Architecture of Modular Intelligence: Project Granite Switch — Modular AI architectures, like Granite-Switch, enable cost-effective LLM deployments by treating adapters as reusable software components.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Niklas Heidloff.