Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge

2026-04-20 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Internet of Things (IoT) & Connected Devices · Depth: Expert, extended

Summary

FLAME (Frequency-aware Latency Analysis for Mobile Edge) is a novel framework designed to accurately estimate AI model inference latency on mobile edge devices, such as NVIDIA Jetson series, under Dynamic Voltage and Frequency Scaling (DVFS). Traditional static profiling methods are rendered invalid by DVFS, and exhaustive profiling is prohibitively expensive, taking days for Small Language Models (SLMs) with variable context lengths. FLAME addresses this by employing a layer-wise modeling approach that quantifies asynchronous CPU-GPU coupling and aggregates dynamic pipeline bubbles across the full model. This bottom-up design generalizes across diverse models, from Deep Neural Networks (DNNs) to SLMs, and significantly reduces profiling overhead from hours/days to mere minutes, while maintaining estimation errors below 8.14%. The framework also enables a deadline-aware DVFS strategy, which outperforms state-of-the-art learning-based approaches like zTT in power efficiency by 23.48% and latency guarantees by 4.35%.

Key takeaway

For MLOps engineers deploying AI models on mobile edge devices with DVFS, FLAME offers a robust solution to overcome the limitations of static profiling and the prohibitive cost of exhaustive methods. You should consider integrating FLAME's layer-wise and model-wise estimation to achieve accurate, frequency-aware latency predictions. This enables more efficient resource management and allows for the implementation of deadline-aware DVFS strategies, significantly improving power efficiency and ensuring latency guarantees for time-critical applications.

Key insights

Precise latency estimation on mobile edge devices requires accounting for dynamic CPU-GPU asynchronous coupling under DVFS.

Principles

• Layer-wise modeling reduces profiling overhead.
• Asynchronous CPU-GPU interaction creates dynamic timing factors.
• Piecewise modeling captures frequency-dependent phase transitions.

Method

FLAME uses layer-wise latency estimation with a piecewise model for CPU-GPU interaction, then aggregates these into a full model timeline, and applies online adaptation for real-time calibration.

In practice

• Profile sparse subsets of frequency combinations.
• Use Hardware Performance Counters (HPCs) for coefficient generalization.
• Implement a decoupled greedy search for DVFS optimization.

Topics

• Mobile Edge Computing
• AI Inference Latency Estimation
• Dynamic Voltage Frequency Scaling
• Asynchronous CPU-GPU Coupling
• Small Language Models

Best for: MLOps Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• GPU Time-Slicing for Concurrent LLM Agents on Kubernetes — GPU time-slicing on Kubernetes hides significant tail latency degradation for latency-sensitive agents, despite stable medians and throughput.
• Beyond Benchmarks: Continuous Edge Inference for Fine-Grained Roadside Perception — Conventional benchmarks significantly overstate real-world edge AI performance due to deployment effects, necessitating temporal stabilization.
• Lambda's MLPerf Inference v6.0: hardware leap, software maturity, research breakthrough — Hardware, software, and routing optimizations significantly boost AI inference performance and reduce latency.
• AI in Multiple GPUs: Understanding the Host and Device Paradigm — Understanding Host-Device interaction, asynchronous execution, and CUDA Streams is fundamental for optimizing GPU performance.
• Micro Language Models Enable Instant Responses — Ultra-compact μLMs enable instant on-device AI responses by initiating text generation while cloud models complete it.
• CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon — CANS optimizes multi-user edge DNN inference by adaptively partitioning models through shared feedback and a novel FedLinUCB-DW algorithm.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.