Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions

2026-04-02 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

HyenaND is a novel subquadratic, global, and input-dependent operator designed for multi-dimensional data, addressing limitations of existing attention and state-space models like Mamba. Unlike Mamba, which rasterizes multi-dimensional inputs into 1D, HyenaND acts directly on native data geometry using implicitly parameterized global, input-dependent multi-dimensional convolutional kernels. Its CUDA implementation, nSubQ, fuses the FFT-convolution path to achieve wall-clock speedups with ℱ(L log L) scaling. Empirical validation across long-context genomics, computer vision (ImageNet), medical imaging (PanTS 3D segmentation), and PDE modeling demonstrates that pure HyenaND stacks match strong attention baselines. Hybrid configurations, interleaving HyenaND and attention layers, consistently outperform both pure attention and recurrence-based hybrids, for instance, achieving a 0.7559 Dice score on PanTS 3D segmentation with 10.8% less peak memory (58.7 GB vs 65.79 GB) than pure attention.

Key takeaway

For AI Architects and Machine Learning Engineers designing models for high-resolution multi-dimensional data, HyenaND offers a compelling alternative to quadratic attention or 1D-rasterizing SSMs. You should consider integrating HyenaND, especially in hybrid configurations with attention, to achieve superior accuracy and memory efficiency on tasks like 3D medical imaging or large-scale vision. This approach allows processing data in its native geometry, avoiding performance-limiting patchification and enabling foundation models to scale to higher resolutions.

Key insights

HyenaND offers a subquadratic, native multi-dimensional operator with input-dependent global convolutions, outperforming 1D-centric alternatives.

Principles

• Multi-dimensional data benefits from native N-dimensional operators.
• Input-dependence can be sample-level, preserving FFT path.
• LTI inductive bias is valuable for natural signals.

Method

HyenaND uses FiLM-modulated SIREN kernels conditioned on register tokens, applied via N-dimensional FFTs. nSubQ fuses FFT-convolution steps for efficiency.

In practice

• Implement HyenaND for N-dimensional data processing.
• Use hybrid HyenaND-attention architectures for performance gains.
• Leverage nSubQ for efficient FFT-convolution operations.

Topics

• Multi-dimensional Data
• Subquadratic Operators
• Convolutional Neural Networks
• FFT Convolutions
• CUDA Optimization
• Medical Imaging
• Genomics

Code references

• NVIDIA/cutlass

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Revisiting Neural Processes via Fourier Transform and Volterra Series — Combining Volterra series with frequency-domain convolutions yields scalable, interpretable translation-equivariant Neural Processes for irregular data.
• Beyond Uniform Sampling: Synergistic Active Learning and Input Denoising for Robust Neural Operators — Combining active learning with input denoising synergistically improves neural operator robustness against adversarial attacks.
• Olmo Hybrid and future LLM architectures — Non-attention architectures are challenging Transformers by offering better scaling and efficiency for long-context LLMs.
• Structural Kolmogorov-Arnold Convolutions: Learnable Function on the Values or the Filter Shape as Parameter-Efficient Alternative to Per-Edge Convolutional KANs — Structural KANs achieve parameter efficiency and strong performance by applying learnable functions to convolution structure or filter shape.
• Unpaired Joint Distribution Modeling via Multi-Scale Image Representations — LUD-MSR models joint distributions from marginals by balancing domain consistency and information preservation via multi-scale representations.
• A Hybrid LSTM--Vision Transformer Architecture for Predicting HRRR Forecast Errors — Hybrid LSTM-ViT models improve weather forecast error prediction by integrating temporal surface data with vertical atmospheric profiles.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.