LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning

2026-06-11 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

LongSpike, a new Spiking Neural Network (SNN) framework, addresses the "memoryless" bottleneck in existing SNN architectures that rely on first-order Ordinary Differential Equations. Developed by integrating fractional-order State-Space Modeling (f-SSM) from control theory, LongSpike extends traditional integer-order SSMs into the fractional-calculus regime. This innovation enables the hierarchical integration of neuronal dynamics with long-memory kernels, significantly improving the model's capacity to capture complex, long-range dependencies in sequential data. To manage the computational overhead associated with fractional operators, LongSpike employs a state-space formulation that supports efficient, parallel training. Empirical evaluations, published on 2026-06-11, demonstrate that LongSpike surpasses state-of-the-art SNNs in accuracy across challenging benchmarks like Long Range Arena (LRA), large-scale WikiText-103, and Speech Commands, all while maintaining sparse synaptic computation. The code is available at https://github.com/xinruihe389-commits/LongSpike.

Key takeaway

For Machine Learning Engineers developing Spiking Neural Networks for long sequence learning, LongSpike presents a significant architectural advancement. You should investigate integrating fractional-order State-Space Modeling into your SNN designs to overcome the "memoryless" bottleneck of first-order ODEs. This approach demonstrably improves accuracy on benchmarks like LRA and WikiText-103 while preserving sparse synaptic computation, offering a path to more powerful and efficient SNNs.

Key insights

LongSpike integrates fractional-order State-Space Modeling into SNNs to capture long-range dependencies efficiently.

Principles

• First-order SNNs create a "memoryless" bottleneck.
• Fractional-order SSMs integrate long-memory kernels.
• Efficient parallel training is crucial for fractional operators.

Method

LongSpike integrates fractional-order State-Space Modeling (f-SSM) into SNNs by extending integer-order SSMs to the fractional-calculus regime. It uses a state-space formulation to enable efficient, parallel training of fractional operators.

In practice

• Apply to Long Range Arena tasks.
• Use for large-scale WikiText-103.
• Implement for Speech Commands.

Topics

• Spiking Neural Networks
• Fractional Calculus
• State-Space Models
• Long Sequence Learning
• Parallel Training
• Long Range Arena

Code references

• xinruihe389-commits/LongSpike

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning — LongSpike integrates fractional-order State-Space Modeling into SNNs to efficiently capture long-range dependencies, outperforming prior SNNs.
• Bullet Trains: Parallelizing Training of Temporally Precise Spiking Neural Networks — Parallel associative scans and differentiable numerical solvers enable efficient, precise training of event-based SNNs.
• An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling — SSMs compute by encoding inputs as traveling waves in a nonlinear oscillator network.
• Improved state mixing in higher-order and block diagonal linear recurrent networks — Richer state mixing in LRNNs enhances expressivity and efficiency for long-sequence modeling.
• Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling — SMET stabilizes dynamic sparse training for LLMs by addressing cold-start issues and optimizing memory, making sparse pre-training practical.
• Fourier Neural Operators with rank-1 lattice points and hyperbolic cross — Replacing FNO's tensor product grids with rank-1 lattices and hyperbolic cross improves efficiency and accuracy using fewer resources.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.