GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature.

2026-07-20 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

GPT-4 employs a Mixture-of-Experts (MoE) architecture, featuring a parameter dictionary totaling 1.8 trillion parameters. When processing each token, the model activates approximately 36 billion parameters, representing only 2% of its total capacity, with the remaining 98% dormant. This selective activation is an intentional design choice, not a limitation, and is supported by a precise mathematical justification. In an MoE layer, the standard transformer feedforward block is replaced by a set of expert networks, and a routing function dynamically selects k of these experts for each token. The layer's output is formally computed as y = Σᵢ∈Top-k(x) G(x)ᵢ · Eᵢ(x), where x is the input token, Eᵢ(x) is expert i's output, G(x)ᵢ is its gating weight, and Top-k(x) identifies the highest-scoring experts.

Key takeaway

For AI Architects evaluating large language model deployments, understanding GPT-4's Mixture-of-Experts architecture is crucial. This design, activating only 2% of its 1.8 trillion parameters per token, implies significant efficiency gains for inference at scale. You should consider how sparse activation models can optimize resource utilization and reduce computational costs compared to dense models, influencing your infrastructure planning and operational strategies.

Key insights

GPT-4's Mixture-of-Experts architecture selectively activates 2% of its 1.8 trillion parameters per token, optimizing scale.

Principles

• MoE replaces standard feedforward layers.
• Routing functions select experts per token.
• Selective activation is a feature, not a bug.

Method

The MoE layer computes y = Σᵢ∈Top-k(x) G(x)ᵢ · Eᵢ(x), where a gating function G(x)ᵢ selects k experts Eᵢ(x) based on input x for each token.

Topics

• GPT-4
• Mixture-of-Experts
• Large Language Models
• Sparse Activation
• Transformer Architecture
• Model Scaling

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature. — GPT-4's 1.8 trillion parameter dictionary uses a sparse Mixture-of-Experts architecture, activating only 2% per token by design.
• GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. — Mixture of Experts (MoE) enables large models to activate only a fraction of parameters per token, boosting efficiency.
• OpenAI launches GPT-5.4 Thinking and Pro combining coding, reasoning, and computer use in one model — GPT-5.4 unifies coding, reasoning, and computer operation, significantly boosting performance in professional knowledge work and desktop navigation.
• Mixture of Experts (MoE) - More Parameters, Same Compute — Mixture of Experts scales model capacity by having many specialized networks, but keeps per-token compute low by activating only a few.
• Kimi K3 Explained: 2.8 Trillion Parameters, 16 Active Experts, 1 Huge AI Shift — Kimi K3 redefines AI scale by using sparse activation of 16 experts from 896, shifting focus to routing and memory.
• GPT 5.5 Arrives, DeepSeek V4 Drops, and the Compute War Intensifies — Domain-specific training and cost-efficiency are becoming critical differentiators for new large language models amidst compute scarcity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.