Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

2026-07-05 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A novel deep neural network compression method, "Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests," addresses hidden-state redundancy by treating trained networks as depth-indexed nonlinear dynamical systems. This framework constructs data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians. It employs A/B/C tests to estimate layer-wise reachable, observable, and jointly reachable-observable ranks, which serve as both diagnostic measures and actual compressed layer widths. Experiments on MNIST demonstrated a four-layer SiLU DNN reduced from 1024 to 277 state order, achieving 72.95% state and 73.48% parameter compression while retaining 95.45% accuracy from an original 96.60%. On CIFAR-10, a larger SiLU DNN saw a reduction from 4608 to 1339 state order, yielding 70.94% state and 83.09% parameter compression, preserving accuracy at 54.44% from 54.45%, and reducing CUDA inference latency by approximately 3X. The approach offers a principled empirical minimal-realisation criterion for compact neural architectures with minimal accuracy loss.

Key takeaway

For Machine Learning Engineers optimizing deep neural network deployment, you should consider the empirical minimal-realisation compression framework. This method allows you to significantly reduce model state order and parameter count, achieving over 70% compression and approximately 3X faster CUDA inference, while preserving accuracy. This provides a principled approach to design compact architectures, directly impacting your resource efficiency and deployment costs.

Key insights

Deep neural network compression can be achieved by applying controllability-observability tests to identify and reduce redundant internal states.

Principles

• Hidden-state redundancy is common in deep neural networks.
• Networks can be modeled as depth-indexed nonlinear dynamical systems.
• Balanced reachable-observable ranks provide a minimal-realisation criterion.

Method

Construct data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians, then use A/B/C tests to estimate layer-wise ranks for compression.

In practice

• Reduce SiLU DNN state order by over 70% for efficiency.
• Achieve approximately 3X CUDA inference latency reduction.

Topics

• Deep Neural Networks
• Model Compression
• Controllability-Observability
• Dynamical Systems
• State-Order Reduction
• Inference Latency

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity — O-minimal definability ensures finite PAC sample complexity for a wide range of feedforward neural networks.
• Residual Modeling for High-Fidelity Learned Compression of Scientific Data — Tailored residual representations preserve learned compression advantages in high-fidelity scientific data scenarios.
• Neural Network Compression by Approximate Differential Equivalence — Compress neural networks by aggregating functionally similar neurons using differential equivalence, not just pruning weights.
• There Will Be a Scientific Theory of Deep Learning — A scientific theory of deep learning, "learning mechanics," is emerging, focusing on dynamics, aggregate statistics, and quantitative predictions.
• MIT Proved 90% of Every AI Model Is Dead Weight. It Took 8 Years for the Hardware to Catch Up. — Most neural network weights are redundant; efficient subnetworks can be trained directly with modern hardware and tooling.
• An Information-Theoretic Criterion for Efficient Data Synthesis — Synthetic data is effective only when external signals keep the training loop "information-open."

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.