Sharpness Aware Surrogate Training for Spiking Neural Networks

2026-03-20 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Sharpness Aware Surrogate Training (SAST) is a novel method for optimizing Spiking Neural Networks (SNNs) that addresses the "surrogate to hard transfer gap" by applying Sharpness Aware Minimization (SAM) to a smooth surrogate forward SNN. Unlike conventional methods that couple a non-smooth forward model with a biased gradient estimator, SAST optimizes a smooth empirical risk, ensuring an exact training gradient for the auxiliary model. The method provides theoretical guarantees including state stability, input Lipschitz bounds, smoothness of the surrogate objective, and nonconvex convergence for stochastic SAST. Empirically, SAST significantly reduces the transfer gap on datasets like N-MNIST (from 65.7% to 94.7% hard spike accuracy) and DVS Gesture (from 31.8% to 63.3% hard spike accuracy), while maintaining high surrogate forward accuracy and improving corruption robustness. The approach requires careful controls for compute matching and threshold calibration for a comprehensive practical assessment.

Key takeaway

Research Scientists developing Spiking Neural Networks should consider integrating Sharpness Aware Surrogate Training (SAST) into their workflow, especially when aiming for robust hard spike deployment. SAST demonstrably improves hard spike accuracy and reduces the critical surrogate-to-hard transfer gap, which is vital for real-world neuromorphic hardware applications. You should rigorously evaluate SAST against compute-matched and calibration-aware baselines to confirm its efficiency and practical superiority for your specific SNN architectures and datasets.

Key insights

SAST optimizes SNNs by applying SAM to a smooth surrogate forward model, reducing the surrogate-to-hard transfer gap.

Principles

• Smooth forward models enable exact gradient computation.
• Sharpness-aware optimization improves generalization and robustness.
• Explicit stability bounds are crucial for recurrent SNNs.

Method

SAST applies SAM to a surrogate forward SNN, using backpropagation through time to compute exact gradients of a smooth objective. It involves two SAM passes with state resets and independent minibatches for convergence guarantees.

In practice

• Use SAST to reduce the surrogate-to-hard transfer gap in SNN deployments.
• Evaluate both surrogate and hard spike accuracy to quantify transfer gap.
• Consider compute-matched baselines for fair performance comparison.

Topics

• Spiking Neural Networks
• Sharpness Aware Minimization
• Surrogate Gradients
• Neuromorphic Computing
• Transfer Gap Reduction

Best for: Research Scientist, AI Researcher, AI Scientist, Deep Learning Engineer

Related on AIssential

• Attacking the Spike: On the Transferability and Security of Spiking Neural Networks to Adversarial Examples — Adversarial attacks on SNNs are highly dependent on surrogate gradient estimators, and a new method, MDSE, exploits this for cross-model effectiveness.
• Optimizing Deep Learning Models with SAM — Sharpness-Aware Minimization (SAM) improves deep learning model generalizability by addressing overparameterization challenges.
• Bullet Trains: Parallelizing Training of Temporally Precise Spiking Neural Networks — Parallel associative scans and differentiable numerical solvers enable efficient, precise training of event-based SNNs.
• Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling — SMET stabilizes dynamic sparse training for LLMs by addressing cold-start issues and optimizing memory, making sparse pre-training practical.
• S-GBT: Smooth Growth Bound Tensor for Certified Robustness Against Word Substitution Attacks in NLP — S-GBT utilizes second-order sensitivity (curvature) to achieve tighter certified robustness against NLP word substitution attacks.
• Bridging Theory and Practice in Crafting Robust Spiking Reservoirs — Robustness interval quantifies stable high-performance regions in spiking reservoirs, guiding practical hyperparameter tuning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.