WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

2026-07-02 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, quick

Summary

WattGPU introduces two predictive models for mean GPU power draw and Inter-Token Latency (ITL) during Large Language Model (LLM) inference. This approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling and enabling generalization to unseen NVIDIA server-grade GPUs and LLMs. Evaluated on 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs, the power draw model achieved a median absolute percentage error (MAPE) of ≤3.4% for offline and ≤13.5% for server scenarios on unseen GPUs. The latency model achieved ≤8.5% in server mode, both maintaining strong GPU ranking correlations (Kendall τ≥0.76). WattGPU reduces MAPE by approximately 4× on unseen LLM-GPU combinations for server scenarios compared to standard baselines. Its data and code are publicly available at https://github.com/maufadel/wattgpu.

Key takeaway

For AI Architects and Machine Learning Engineers optimizing LLM deployments, WattGPU offers a critical tool to predict inference power and latency on unseen GPUs and LLMs. You can now make informed hardware selection decisions and optimize resource allocation without the time-consuming and costly process of exhaustive hardware profiling. This enables more efficient data center operations and reduced energy consumption for your LLM workloads.

Key insights

WattGPU predicts LLM inference power and latency on new hardware without profiling, using only public metadata.

Principles

• Public metadata enables cross-hardware prediction.
• Generalization to unseen hardware is achievable.
• Predictive models outperform physical baselines.

Method

WattGPU employs two predictive models for mean GPU power draw and Inter-Token Latency, trained on public LLM metadata and GPU specifications, validated via leave-one-GPU/LLM-out cross-validation.

In practice

• Optimize LLM-GPU matching without profiling.
• Estimate energy consumption for new deployments.
• Compare GPU efficiency for LLM workloads.

Topics

• LLM Inference
• GPU Performance Prediction
• Energy Efficiency
• Data Center Optimization
• NVIDIA GPUs
• Predictive Modeling

Code references

• maufadel/wattgpu

Best for: MLOps Engineer, Research Scientist, CTO, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Building a monokernel for LLM inference on AMD MI300X - up to 3,300 output tokens/s per request [P] — Optimizing LLM inference on AMD MI300X by leveraging hardware die topology with a single GPU-resident monokernel.
• Maximize AI Factory Energy Efficiency Through Full-Stack Inference and Training Optimizations — Full-stack optimization, from hardware to models, is critical for maximizing AI factory performance per watt and minimizing token costs.
• PALS: Power-Aware LLM Serving for Mixture-of-Experts Models — PALS optimizes LLM serving energy efficiency by dynamically controlling GPU power caps and batch sizes using feedback-driven models.
• Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization — Parallel Track Transformers reduce GPU synchronization for faster, more efficient LLM inference.
• assigning Moe to Gpus to reduce inference and memory usage — Optimizing MoE expert-to-GPU assignment using training logs is a critical challenge for LLM inference efficiency.
• The Next Frontier: How Speculative Decoding Is Eating the LLM Inference Stack — Speculative decoding boosts LLM inference speed by parallelizing token verification, leveraging idle GPU compute without quality loss.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.