Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A new latency-oriented neural network learning method is proposed to optimize deep neural networks for edge devices, specifically addressing strict latency constraints while maintaining high accuracy. This approach incorporates a universal hardware-customized latency predictor, enabling a one-shot training process. Experiments demonstrate significant improvements: on an NVIDIA Jetson Nano, GoogLeNet's latency was reduced from 40.32 ms to 34 ms (meeting a 34 ms constraint) with only a 0.14% accuracy drop, further improving to 0.04% with quantization. On an NVIDIA Jetson TX2, VGG-19 was compressed from 119.98 ms to 34 ms, improving accuracy by 0.5%, and GoogLeNet scaled from 20.27 ms to 34 ms, boosting accuracy by 0.78%. The framework is open-sourced at https://github.com/ntuliuteam/ZeroBN.

Key takeaway

For AI Engineers deploying models to edge devices with strict real-time latency constraints, this latency-oriented learning method provides a direct solution. It enables optimizing DNNs to meet specific temporal budgets, as demonstrated by reducing GoogLeNet's latency to 34 ms on a Jetson Nano with minimal accuracy impact, or even improving VGG-19's accuracy on a Jetson TX2. You should explore integrating this open-source framework to achieve efficient, high-performance edge deployments.

Key insights

A latency-oriented DNN learning method optimizes edge models for strict real-time constraints with one-shot training.

Principles

• Directly optimize for temporal cost in edge DNNs.
• Hardware-customized predictors enable efficient optimization.

Method

The method uses a latency-oriented neural network learning approach combined with a universal hardware-customized latency predictor to achieve one-shot training for models satisfying specific latency constraints.

In practice

• Reduce GoogLeNet latency to 34 ms on Jetson Nano.
• Compress VGG-19 to 34 ms on Jetson TX2.

Topics

• Edge AI
• Latency Optimization
• Neural Architecture Search
• Zerorized Batch Normalization
• GoogLeNet
• NVIDIA Jetson

Code references

• ntuliuteam/ZeroBN

Best for: Computer Vision Engineer, Research Scientist, Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge — Precise latency estimation on mobile edge devices requires accounting for dynamic CPU-GPU asynchronous coupling under DVFS.
• Training the AIs' Eyes: How Roboflow is Making the Real World Programmable, with CEO Joseph Nelson — Computer vision, though advancing, lags LLMs due to real-world data complexity and demands efficient, task-specific model deployment.
• Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs — Coordinating existing pruning signals through architecture-aware, layer-wise sparsification yields more efficient and reliable DNN compression.
• CANS: Accelerating Multiuser Collaborative Edge Inference via Cooperative Autodidactic NeuroSurgeon — CANS optimizes multi-user edge DNN inference by adaptively partitioning models through shared feedback and a novel FedLinUCB-DW algorithm.
• Accelerating on-device AI: A look at Arm and Google AI Edge optimization — Arm SME2 and Google AI Edge enable efficient, high-quality on-device generative AI inference on CPUs.
• Budget-Adaptive Routing: Skipping the Weak When the Strong Answers Anyway — Budget-Adaptive Routing dynamically selects optimal offloading strategies based on varying computational budgets for edge-cloud inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.