GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature.

2026-07-20 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

GPT-4 utilizes a Mixture-of-Experts (MoE) architecture, which includes a parameter dictionary of 1.8 trillion parameters. For each token processed, GPT-4 activates approximately 36 billion parameters, representing about 2% of its total parameter dictionary. This sparse activation is not a limitation but a deliberate architectural choice with a precise mathematical justification. Unlike standard transformer feedforward blocks where all tokens pass through the same parameters, MoE layers replace this with a set of expert networks, and a routing function selects a subset of these experts for each token. Understanding this design is crucial for deploying such systems at scale.

Key takeaway

For AI Architects and Machine Learning Engineers optimizing large language model deployments, understanding GPT-4's Mixture-of-Experts architecture is critical. Your resource allocation and scaling strategies should account for sparse parameter activation, where only a fraction of the total parameters are active per token. This implies focusing on efficient routing and expert management rather than assuming uniform parameter load, potentially leading to more cost-effective and performant systems.

Key insights

GPT-4's 1.8 trillion parameter dictionary uses a sparse Mixture-of-Experts architecture, activating only 2% per token by design.

Principles

• MoE replaces standard feedforward layers with multiple expert networks.
• A routing function selects k experts for each token.
• Parameter dictionary size differs from active parameters per computation.

Method

The MoE layer computes y = Σᵢ∈Top-k(x) G(x)ᵢ · Eᵢ(x), where x is input, Eᵢ(x) is expert output, G(x)ᵢ is gating weight, and Top-k(x) selects k experts.

In practice

• Rethink large model deployment strategies.
• Optimize for sparse activation patterns.

Topics

• GPT-4
• Mixture-of-Experts
• Large Language Models
• Parameter Efficiency
• Model Architecture
• Sparse Activation

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature. — GPT-4's Mixture-of-Experts architecture selectively activates 2% of its 1.8 trillion parameters per token, optimizing scale.
• GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. — Mixture of Experts (MoE) enables large models to activate only a fraction of parameters per token, boosting efficiency.
• Understanding Mixture of Experts (MoE): How Modern LLMs Scale to Trillions of Parameters — Mixture of Experts enables LLMs to scale to trillions of parameters by activating only relevant sub-networks per token.
• OpenAI launches GPT-5.4 with Pro and Thinking versions — GPT-5.4 offers enhanced capabilities, efficiency, and safety features for professional AI applications.
• Kimi K3 Explained: 2.8 Trillion Parameters, 16 Active Experts, 1 Huge AI Shift — Kimi K3 redefines AI scale by using sparse activation of 16 experts from 896, shifting focus to routing and memory.
• Mixture of Experts (MoE) - More Parameters, Same Compute — Mixture of Experts scales model capacity by having many specialized networks, but keeps per-token compute low by activating only a few.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.