The Memory in Your Thumb Drive Could Fix AI’s Big Problem

2026-07-14 · Source: IEEE Spectrum · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Emerging Technologies & Innovation · Depth: Intermediate, short

Summary

High Bandwidth Flash (HBF) is an emerging memory technology designed to address the immense memory demands of Large Language Models (LLMs), particularly for AI inference workloads. Unlike traditional NAND flash, HBF employs 3D packaging and vertical stacking techniques, similar to High Bandwidth Memory (HBM), to significantly boost bandwidth. While standard NAND flash offers up to 4.8GB/s per die, and HBM4E reaches 3.6TB/s per stack, Sandisk projects its first-generation HBF to stack up to 16 NAND chips for 512GB capacity and 1.6TB/s read bandwidth, with future generations targeting 2TB/s and 3.2TB/s. HBF is optimized for read-heavy AI inference, where frozen model weights can be stored, freeing HBM for high-speed scratchpad operations. SK Hynix and Sandisk launched a joint effort on February 25, 2026, to standardize HBF within the Open Compute Project (OCP), aiming to reduce accelerator requirements, improve energy efficiency, and lower data center costs.

Key takeaway

For AI Architects and Machine Learning Engineers designing inference systems, consider integrating High Bandwidth Flash (HBF) into your memory hierarchy. This technology can offload static model weights and KV caches from expensive HBM, significantly reducing the number of accelerators needed and improving energy efficiency. As HBF standardization progresses, planning for its adoption could lead to substantial cost savings and enable greater scaling of your AI inference hardware.

Key insights

High Bandwidth Flash (HBF) offers a cost-effective, high-capacity memory solution for read-heavy AI inference workloads.

Principles

• Stacking NAND flash dies significantly increases memory bandwidth and capacity.
• Non-volatile memory is suitable for static, read-only AI model weights.
• Complementary memory tiers can optimize performance and cost in AI systems.

Method

HBF improves bandwidth by applying advanced 3D packaging and vertical stacking to NAND flash dies, creating a memory-dense chip. This allows it to house massive, read-heavy data like frozen model weights.

In practice

• Utilize HBF for storing static multi-billion parameter model weights.
• Employ HBF for precomputed KV caches in inference environments.

Topics

• High Bandwidth Flash
• NAND Flash Memory
• AI Inference
• Large Language Models
• 3D Packaging
• Open Compute Project

Best for: MLOps Engineer, AI Engineer, CTO, AI Hardware Engineer, AI Architect, Machine Learning Engineer

Related on AIssential

• NAND Reimagined in High-Bandwidth Flash to Complement HBM — HBF stacks NAND dies for high-capacity, high-throughput memory, complementing HBM in AI inference architectures.
• Massive AI Storage Demand Creates a New Memory Wall — AI's massive memory demands necessitate new architectures like high-bandwidth flash, optimizing capacity and sequential bandwidth over traditional DRAM.
• Silicon Built for One Thing: How Custom AI Hardware Is Smashing the Throughput Ceiling — The core bottleneck in AI inference is memory bandwidth, not just compute, driving custom hardware innovation.
• Stop adding more GPUs: Weka's new storage platform reduces load by caching 100% of an AI model's pre-calculated tokens — Extending GPU memory with cheaper flash storage significantly reduces AI inference costs and improves utilization.
• [Server] The Bottleneck of Compute: How Humanity Weaves the Soul of Memory with Silicon — HBM overcomes the "Memory Wall" by prioritizing wide parallel data paths and 3D stacking over frequency.
• Breaking the Memory Wall in the Age of Inference — Integrating compute directly into SRAM cells via Digital In-Memory Compute (DIMC) effectively addresses AI inference memory bottlenecks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by IEEE Spectrum.