Principled approaches for extending neural architectures to function spaces for operator learning

2026-07-03 · Source: Nature Machine Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This article outlines principled approaches for extending neural architectures to function spaces, focusing on neural operators for scientific problems like those governed by partial differential equations. It distills key design principles for creating practical neural operators that map input functions to output functions, rather than finite-dimensional vectors. The authors propose a systematic recipe for converting existing neural network layers, such as fully connected layers, convolutions, graph neural networks, and transformers, into their function-space operator analogues. Empirical evaluations using Navier-Stokes equations demonstrate that architectures like FNO and OFORMER, designed with domain-level receptive fields and principled discretization, exhibit strong generalization across varying resolutions, unlike fixed-discretization networks such as U-Net and ViT. The work emphasizes the importance of coordinate-aware continuous operators and quadrature weights for robust cross-resolution performance.

Key takeaway

For Machine Learning Engineers developing models for scientific simulations or PDEs, you should prioritize neural operator architectures. Standard neural networks like U-Net or ViT degrade significantly when tested on resolutions different from training data. Instead, adopt neural operator design principles, which ensure discretization-agnostic models like FNO and OFORMER generalize robustly across varying resolutions, enabling more reliable and flexible scientific machine learning applications.

Key insights

Neural operators extend deep learning to function spaces, enabling discretization-agnostic learning for scientific applications.

Principles

• Discretization agnostic: consistent outputs across resolutions.
• Fixed number of parameters, independent of discretization.
• Universal approximation of sufficiently regular operators.

Method

Convert neural network layers by identifying their continuous operator analogue, then discretize it using coordinate-dependent functions and replacing sums with quadrature-weighted integrals to ensure discretization agnosticism.

In practice

• Achieve zero-shot super-resolution.
• Improve data efficiency with multi-resolution training.
• Use NEURALOPERATOR library for implementation.

Topics

• Neural Operators
• Function Spaces
• Operator Learning
• Partial Differential Equations
• Discretization Agnosticism
• Deep Learning Architectures

Code references

• neuraloperator/neuraloperator

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Operator Learning for Smoothing and Forecasting — A new theory underpins data-driven smoothing and forecasting in dynamical systems using neural operators.
• PINNs and Neural Operators: Two Competing Visions of Scientific AI — PINNs solve specific PDE instances via physics-informed loss, while Neural Operators learn general solution mappings.
• Reformulating Neural Operators in $d+1$ Dimensions for Embedding Evolution — Neural operators redefined in d+1 dimensions using Schrödingerisation better capture system evolution and achieve superior performance.
• Learning Physical Operators using Neural Operators — A physics-informed framework improves neural operator generalization and temporal continuity for PDE solving via operator splitting.
• Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learning — Shared representations in multi-task operator learning achieve near-optimal rates without increasing overall cost.
• Stability and Discretization Error of State Space Model Neural Operators — Theoretical bounds quantify discretization error and stability in neural operators for solving PDEs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Nature Machine Intelligence.