Powering the AI Inference Wave with EPRI's Ben Sooter - Ep. 292

2026-03-04 · Source: NVIDIA AI Podcast · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, AI Grid Integration · Depth: Intermediate, extended

Summary

EPRI's Director of R&D, Ben Sooter, discusses the evolving relationship between AI data centers and the electric grid, highlighting that approximately 80% of an AI model's lifetime energy consumption occurs during inference, not training. He introduces the concept of micro data centers, which are smaller, geographically dispersed facilities designed to handle inference loads closer to end-users. These micro data centers, typically 3-20 megawatts, can be strategically located near underutilized electrical substations to leverage existing infrastructure, enhance grid resilience, and reduce interconnection queues. This distributed approach aims to meet the growing demand for low-latency AI services while optimizing energy distribution and potentially integrating with clean energy solutions like solar, wind, and energy storage.

Key takeaway

For CTOs and VPs of Engineering planning AI infrastructure, recognize that inference will dominate your long-term energy costs and demand patterns. You should investigate distributed micro data center strategies, particularly co-locating with underutilized substations, to ensure low-latency service delivery and enhance grid stability, avoiding the bottlenecks and high costs associated with large, centralized training facilities.

Key insights

AI model inference consumes 80% of lifetime energy, necessitating distributed micro data centers for grid resilience and low-latency services.

Principles

• Inference drives most AI energy demand.
• Geographic proximity improves AI service latency.
• Utilize existing grid infrastructure for efficiency.

Method

Deploy micro data centers (3-20 MW) near underutilized substations, distributing compute capacity across multiple sites to meet regional demand and optimize grid integration.

In practice

• Consider micro data centers for latency-sensitive AI applications.
• Explore substation co-location for new data center builds.
• Integrate energy storage for demand flexibility.

Topics

• AI Energy Demand
• Micro Data Centers
• AI Inference
• Electric Grid Integration
• Distributed Inference

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Machine Learning Engineer, MLOps Engineer, AI Operations Specialist

Related on AIssential

• Your Next AI Query May Travel Where the Power Is — Distributing AI inference workloads across micro data centers near substations enhances grid flexibility and power access.
• Maciel’s article is right to puncture the “AI is magic intelligence” narrative and re-anchor the debate in infrastructure, energy, and economics. But the chapter after that will be written by those... — AI's future is an infrastructure and economics contest, heavily influenced by governance and security.
• Why Data Centers Must Become System‑Aware — Data centers must evolve into system-aware infrastructure, integrating with energy and communication networks to meet AI and 5G demands.
• Why AI’s Deployment Problem May Create the Next Infrastructure Giants — AI's next major bottleneck is deployment and inference, not just model training, driving demand for specialized, distributed infrastructure.
• The newest AI boom pitch: Host a mini data center at your home — Distributed data centers in homes can expand AI compute and reduce infrastructure burdens by leveraging existing residential power capacity.
• The AI Infrastructure Divide — The global AI boom is creating a material and geopolitical divide, concentrating compute power and economic advantage in a few wealthy regions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA AI Podcast.