TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers

2026-07-21 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

TEFormer is a novel Spiking Transformer framework designed to enhance temporal modeling by implementing structured bidirectional temporal fusion. Inspired by the human visual pathway, TEFormer decouples temporal processing into two core components: a lightweight, hyperparameter-free Temporal Enhanced Attention (TEA) module for parallel forward temporal fusion in the attention mechanism, and a backward gated recurrent Temporal MLP (T-MLP) for reverse-order temporal information aggregation. This approach addresses limitations in existing Spiking Transformers' ability to exploit spatiotemporal dependencies. Extensive experiments demonstrate TEFormer's consistent and significant outperformance against strong SNN and Spiking Transformer baselines across static (e.g., >96% on CIFAR10), neuromorphic (e.g., 81.90% on CIFAR10-DVS, 95.95% on NCARS), and temporally complicated datasets. Its performance gains remain stable across various neural encoding schemes, establishing TEFormer as an effective and general framework for temporal modeling.

Key takeaway

For Machine Learning Engineers developing Spiking Neural Networks for sequence modeling, you should consider implementing TEFormer's structured bidirectional temporal fusion. This approach, which decouples forward and backward temporal processing, consistently improves performance across static, neuromorphic, and complex temporal datasets, even with varied neuron encoding schemes. Adopting this method can significantly enhance your SNNs' ability to capture spatiotemporal dependencies efficiently, offering a robust path to higher accuracy in energy-constrained applications.

Key insights

TEFormer introduces structured bidirectional temporal fusion to Spiking Transformers, significantly improving spatiotemporal dependency exploitation.

Principles

• Bidirectional temporal fusion enhances Spiking Transformer accuracy.
• Decoupling temporal modeling improves computational efficiency.
• Learnable temporal decay parameters adapt better than fixed values.

Method

TEFormer integrates Temporal Enhanced Attention (TEA) for parallel forward fusion using a single learnable scalar and a temporal mask matrix. It also uses a Temporal MLP (T-MLP) with a gated recurrent update for backward temporal tracing, starting from the last time step.

In practice

• Design distinct forward and backward temporal modules.
• Employ learnable scalar parameters for temporal decay.
• Test SNNs with various neuron encoding schemes.

Topics

• Spiking Neural Networks
• Spiking Transformers
• Temporal Modeling
• Bidirectional Fusion
• Neuromorphic Computing
• Attention Mechanisms

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• PSViT: A Methodology for Structurally Pruning Spiking Vision Transformers — Structured pruning of SViT models enables efficient deployment on existing hardware, overcoming limitations of unstructured methods.
• Three factor delay learning rules for spiking neural networks — Online learning of synaptic and axonal delays significantly enhances SNN performance and efficiency for temporal tasks.
• Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformer — Otters++ employs physical hardware signal decay for energy-efficient time-to-first-spike computation in optical spiking Transformers.
• LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning — LongSpike integrates fractional-order State-Space Modeling into SNNs to efficiently capture long-range dependencies, outperforming prior SNNs.
• S3T-Former: A Purely Spike-Driven State-Space Topology Transformer for Skeleton Action Recognition — S3T-Former is a purely spike-driven Transformer for energy-efficient skeleton action recognition, achieving high accuracy with extreme sparsity.
• PMSN: A Parallel Multi-compartment Spiking Neuron for Multi-scale Temporal Processing — PMSN enhances SNNs' multi-scale temporal processing and training speed by emulating biological multi-compartment neurons with parallelization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.