Pathwise Learning of Stochastic Dynamical Systems with Partial Observations

2026-06-24 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Data Science & Analytics · Depth: Expert, extended

Summary

This work presents a neural path estimation approach for pathwise learning of stochastic dynamical systems from partial observations. It tackles the complex problem of simultaneously approximating Stochastic Differential Equation (SDE) coefficients and inferring posterior updates from noisy, indirect measurements. The method formulates a stochastic control problem to solve filtering posterior path measures, which correspond to a pathwise Zakai equation. A generative model is then constructed to map prior path measures to posterior measures using controlled diffusion and a Radon-Nikodym derivative. The approach learns the control by embedding noisy observation paths, allowing for the recovery of conditional path measures and associated SDEs. This sidesteps issues like particle degeneracy common in traditional methods. Experiments on nonlinear dynamical systems, including double-well, Lorenz 63, and Lorenz 96, demonstrate its effectiveness in learning multimodal, chaotic, and high-dimensional systems.

Key takeaway

For AI Scientists and Machine Learning Engineers inferring complex stochastic dynamics from noisy, partial observations, this method offers a robust alternative to traditional particle filters. You can generate accurate data-assimilated trajectories and quantify uncertainty for path functionals without expensive iterative updates. Consider applying this neural path estimation approach to systems with multimodal, chaotic, or high-dimensional characteristics, especially when dealing with missing or non-equidistant observations.

Key insights

A neural path estimation approach uses stochastic control and variational inference to learn SDEs from noisy, partial observations.

Principles

• Pathwise filtering addresses inherent path dependence.
• Optimal control minimizes KL divergence for path measures.
• Amortized inference avoids re-training for new observations.

Method

Derive a stochastic control problem from the pathwise Zakai equation. Construct a generative model mapping prior to posterior path measures via controlled diffusion, learning control through noisy observation path embeddings.

In practice

• Generate data-assimilated trajectories for new observations.
• Perform uncertainty quantification for path functionals.
• Handle missing observations and non-equidistant time steps.

Topics

• Pathwise Filtering
• Stochastic Control
• Neural SDEs
• Data Assimilation
• Variational Inference
• Generative Models

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Relative Entropy Estimation in Function Space: Theory and Applications to Trajectory Inference — A new functional KL divergence estimator provides a principled way to evaluate trajectory inference under partial observability.
• Closing the Approximation Gap in Simulation-free Latent SDEs — Simulation-free VI for latent SDEs can restrict posterior path laws, degrading inference; Helmholtz-SDE resolves this.
• A Stabilized Path-Space Approach to Diffusion-Based Posterior Sampling — A stabilized path-space framework improves diffusion posterior sampling by connecting it to stochastic optimal control and removing bias.
• Robust Parameter and State Estimation in Multiscale Neuronal Systems Using Physics-Informed Neural Networks — PINNs offer robust parameter and state estimation in multiscale neuronal systems, overcoming limitations of traditional methods.
• Two-Layer Linear Auto-Regressive Models Estimate Latent States — Two-layer linear auto-regressive models approximate Kalman filtering for latent state estimation in partially observed linear dynamical systems.
• Bayesian 3D Steerable CNNs: Enabling Equivariance and Uncertainty Quantification Simultaneously — Bayesian Steerable-CNNs unify SE(3)-equivariance with uncertainty quantification via stochastic kernels and variational inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.