Neural Legendre-Fenchel transform with Hessian Preconditioning

2026-06-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

A new method, Neural Legendre-Fenchel transform with Hessian Preconditioning, addresses challenges in approximating convex conjugates of ill-conditioned functions. Building on the Legendre-Fenchel (LF) transform's reformulation as projective polarity, this work introduces a Hessian-based preconditioning strategy. It leverages affine invariance by applying an affine deformation around a minimizer, making the second-order Taylor approximation coincide with a canonical paraboloid. A residual network then learns this simplified mapping, with the original conjugation recovered via inverse deformation. This approach incurs modest overhead: a single eigendecomposition during initialization and two matrix-vector multiplications per query. Experiments show improved convergence rates and enhanced numerical accuracy, especially for high-dimensional and ill-conditioned problems.

Key takeaway

For research scientists developing numerical methods for convex analysis, this Hessian-preconditioning technique offers a robust solution for ill-conditioned functions. You can achieve significantly improved convergence and numerical accuracy in approximating convex conjugates. Consider integrating this affine invariance-based preconditioning, noting its modest computational overhead of one eigendecomposition and two matrix-vector multiplications per query.

Key insights

Hessian-based preconditioning improves neural Legendre-Fenchel transform accuracy for ill-conditioned functions.

Principles

• Leverage affine invariance for function simplification
• Initialize residual networks near identity mapping

Method

Apply affine deformation to align a function's second-order Taylor approximation with a canonical paraboloid, then use a residual network, and inverse deform.

In practice

• Enhance conjugation accuracy for ill-conditioned problems
• Improve convergence rates in high-dimensional benchmarks

Topics

• Legendre-Fenchel Transform
• Convex Analysis
• Neural Networks
• Hessian Preconditioning
• Affine Invariance
• Numerical Accuracy

Best for: AI Scientist, Research Scientist

Related on AIssential

• Consistency Regularised Gradient Flows for Inverse Problems — A new gradient-flow framework improves LDM-based inverse problem solving by reducing NFEs and backpropagation.
• PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training — Polynomial preconditioning stabilizes LLM training by controlling weight singular values, incurring no inference overhead.
• Generalization in Nonlinear Least Squares via Learned Feature Geometry — New generalization bounds for nonlinear least-squares models depend on learned feature geometry and data-dependent effective dimension, not just parameter count.
• Gradient Regularized Newton Boosting Trees with Global Convergence — A gradient-regularized Newton boosting scheme achieves global convergence for GBDTs, addressing prior theoretical gaps.
• JA-SIREN: Deterministic Initialization for Sinusoidal Networks via Spectral Matching — Deterministic initialization via spectral matching for sinusoidal networks significantly enhances INR reproducibility and performance.
• Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval — Hopfield-Fenchel-Young networks unify and extend associative memory models through generalized energy functions and differentiable update rules.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.