Stacking Chips Sideways Gives AI More Memory

2026-07-08 · Source: IEEE Spectrum · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation, Semiconductor & Chip Design · Depth: Expert, short

Summary

Two research groups presented novel memory architectures at the IEEE VLSI Symposium, addressing the thermal and bandwidth limitations of High-Bandwidth Memory (HBM) in AI accelerators. Current HBM, like 12-die stacks delivering 36 gigabytes per stack and 2800 GB/s with HBM4, faces overheating and scaling challenges. South Korean researchers introduced V-Die, a side-stacked design with microfluidic cooling channels maintaining 45℃, significantly cooler than typical 80℃+ peaks. V-Die eliminates through-silicon vias (TSVs) and integrates individual I/O systems, offering four times more connections than HBM4 and a 37 percent faster read time. Simulations predict an 82 percent speed boost over HBM4, achieving 540 tokens per second for GPT3-sized models, and a 32 percent latency reduction. Separately, Japanese researchers presented MOSAIC, which uses inductive coupling for flexible die connections. MOSAIC fits 98 dies per cube, providing 294 GB of memory, with a peak temperature of 81.3℃, with potential for 882 GB by thinning dies.

Key takeaway

For AI Hardware Engineers designing next-generation accelerators, these side-stacked memory innovations are critical. You should evaluate V-Die's microfluidic cooling and TSV-free design for significant speed and latency improvements, potentially boosting GPT3-sized model performance by 82 percent. Alternatively, consider MOSAIC's inductive coupling for high-capacity solutions, offering up to 882 GB per cube, to address memory bottlenecks without substantial thermal increases. Your architectural decisions now will dictate future AI model scalability.

Key insights

Side-stacked memory designs like V-Die and MOSAIC address HBM's thermal and bandwidth limitations for AI accelerators.

Principles

• Vertical HBM stacking creates thermal resistance.
• Through-silicon vias consume memory area.
• Inductive coupling enables flexible die integration.

Method

V-Die involves vertical DRAM stacking with microfluidic cooling and edge I/O. MOSAIC uses inductive coupling for side-stacked die-to-substrate connections, with power on cube sides.

In practice

• V-Die offers 82% speed increase over HBM4.
• MOSAIC can double HBM4 capacity to 294 GB.
• Thinning dies enables 882 GB MOSAIC cubes.

Topics

• High-Bandwidth Memory
• AI Accelerators
• Memory Architecture
• Thermal Management
• V-Die
• MOSAIC

Best for: AI Hardware Engineer, AI Scientist, Research Scientist

Related on AIssential

• IBM Shows Sub-1-nm Chips, Targeting Production in 5 Years — IBM's "nanostack" technology enables sub-1-nm chips with unprecedented transistor density and SRAM scaling for future AI processors.
• [Server] The Bottleneck of Compute: How Humanity Weaves the Soul of Memory with Silicon — HBM overcomes the "Memory Wall" by prioritizing wide parallel data paths and 3D stacking over frequency.
• Rebellions Bets on Memory-Centric Architecture as It Weighs IPO Options — Memory-centric AI architectures and supply chain security are critical for large-scale LLM inference and market traction.
• SK hynix Ships Samples of 12-Layer Next-Gen ‘HBM4E’ — SK hynix's HBM4E samples demonstrate significant advancements in AI memory performance, power efficiency, and thermal management.
• Apple's M5 Pro & Max Architecture is INSANE — Apple's M5 Fusion architecture uses asymmetric dual-die design for scalable performance and efficiency in portable devices.
• Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480] — Etched's architecture leverages low voltage inference and cluster scale memory to achieve significantly faster and cheaper AI model inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by IEEE Spectrum.