A Hitchhiker's Guide to Poisson Gradient Estimation

2026-02-05 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computational Neuroscience · Depth: Expert, extended

Summary

This paper introduces a modified Exponential Arrival Time (Eat) method, termed Eat${}_{ extsf{cubic}}$, for differentiating through discrete Poisson-distributed latent variables in models like Variational Autoencoders (VAEs) and Partially Observable Generalized Linear Models (POGLMs). The original Eat method, Eat${}_{ extsf{sigmoid}}$, and the Gumbel-SoftMax (Gsm) relaxation struggle with hyperparameter sensitivity and distributional fidelity. Eat${}_{ extsf{cubic}}$ replaces the sigmoid approximation with a cubic Hermite interpolant, which has compact support, theoretically guaranteeing an unbiased first moment and reducing second-moment bias. Empirical evaluations across distributional fidelity (Wasserstein-1 distance), gradient quality (BiasEnergy, NoiseEnergy, Cosine Similarity), and performance on $\operatorname{\mathcal{P}}$-VAE and POGLM tasks demonstrate that Eat${}_{ extsf{cubic}}$ consistently outperforms Eat${}_{ extsf{sigmoid}}$ and Gsm, exhibiting superior robustness to temperature hyperparameter choices and often matching exact gradients.

Key takeaway

For research scientists developing or applying Poisson latent variable models, you should adopt the new Eat${}_{ extsf{cubic}}$ relaxation method. Its superior distributional fidelity and robustness to temperature hyperparameter choices will significantly reduce the need for costly grid searches and lead to more stable and reliable training, ultimately improving model performance and generalization in NeuroAI applications.

Key insights

Eat${}_{ extsf{cubic}}$ offers a robust, unbiased method for Poisson gradient estimation by using cubic smoothstep.

Principles

• Compact support improves distributional fidelity.
• Distributional fidelity and gradient quality are complementary metrics.
• Temperature hyperparameter robustness is critical for practical utility.

Method

The Eat${}_{ extsf{cubic}}$ method replaces the sigmoid function in the Exponential Arrival Time (Eat) relaxation with a cubic Hermite interpolant (smoothstep) to ensure compact support, thereby reducing bias in moment estimation and improving temperature robustness.

In practice

• Use Eat${}_{ extsf{cubic}}$ for Poisson latent variable models.
• Prioritize methods with robust temperature performance.
• Evaluate both distributional fidelity and gradient quality.

Topics

• Poisson Gradient Estimation
• Latent Variable Models
• Variational Autoencoders
• Gumbel-SoftMax
• Computational Neuroscience

Code references

• hadivafaii/PoissonGradientEstimation

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• A Differentiable Bayesian Relaxation for Latent Partial-Order Inference — A differentiable Bayesian relaxation enables scalable inference of latent partial orders from linear data by smoothing discrete constraints.
• Robust Parameter and State Estimation in Multiscale Neuronal Systems Using Physics-Informed Neural Networks — PINNs offer robust parameter and state estimation in multiscale neuronal systems, overcoming limitations of traditional methods.
• Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization — Connecting stochastic resetting to ridge regression offers a renewal perspective on spectral regularization.
• Semiparametric Efficient Bilevel Gradient Estimation — Semiparametric debiasing removes first-order bias in functional bilevel gradient estimation, yielding asymptotically normal estimators.
• Deep Doubly Debiased Longitudinal Effect Estimation with ICE G-Computation — D³-Net combines SDR-based training with LTMLE targeting to robustly estimate longitudinal treatment effects, mitigating error propagation.
• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network — A new model tracks average trajectory and oscillation covariance, revealing an effective free energy for Edge of Stability dynamics.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.