Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics

2026-04-23 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Mathematics & Computational Sciences · Depth: Expert, medium

Summary

Heng Yang proposes a sampling-based framework called Tempered Sequential Monte Carlo (TSMC) for finite-horizon trajectory and policy optimization, detailed in a paper published April 23, 2026. This method casts controller design as an inference problem, minimizing a KL-regularized expected trajectory cost to derive an optimal "Boltzmann-tilted" distribution over controller parameters. TSMC employs an adaptive annealing scheme that reweights and resamples particles along a tempering path, moving from a prior to a sharp, potentially multimodal target distribution. It integrates Hamiltonian Monte Carlo rejuvenation to maintain diversity and leverages exact gradients from differentiating through trajectory rollouts. For policy optimization, TSMC extends its capabilities by using a deterministic empirical approximation of the initial-state distribution and an extended-space construction that treats rollout randomness as auxiliary variables. Experimental results on various benchmarks indicate that TSMC is broadly applicable and performs favorably against existing state-of-the-art baselines.

Key takeaway

For research scientists developing control systems with differentiable dynamics, TSMC offers a robust framework for optimizing trajectories and policies. You should consider integrating this sampling-based inference approach, particularly its adaptive tempering and Hamiltonian Monte Carlo rejuvenation, to efficiently explore complex, multimodal solution spaces and achieve superior performance compared to current baselines. This method can lead to more stable and accurate controller designs.

Key insights

TSMC optimizes trajectories and policies by framing controller design as inference, using tempered sampling and gradient-based rejuvenation.

Principles

• Controller design can be framed as an inference problem.
• Adaptive annealing improves sampling from complex distributions.
• Exact gradients enhance Monte Carlo diversity.

Method

TSMC minimizes KL-regularized expected trajectory cost, then uses an annealing scheme with adaptive reweighting and resampling, plus Hamiltonian Monte Carlo rejuvenation, to sample from the resulting Boltzmann-tilted distribution.

In practice

• Apply TSMC for finite-horizon trajectory optimization.
• Use TSMC for policy optimization with differentiable dynamics.
• Leverage exact gradients for improved sampling efficiency.

Topics

• Tempered Sequential Monte Carlo
• Trajectory Optimization
• Policy Optimization
• Differentiable Dynamics
• Hamiltonian Monte Carlo

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Counterdiabatic Hamiltonian Monte Carlo — CHMC accelerates HMC sampling by learning a counterdiabatic term to efficiently transport particles across time-varying distributions.
• Efficient Stochastic Optimisation via Sequential Monte Carlo — SOSMC efficiently optimizes intractable gradient functions by replacing costly MCMC sampling with faster Sequential Monte Carlo approximations.
• Reasoning with Sampling: Cutting at Decision Points — Entropy-guided resampling at decision points significantly improves reasoning efficiency and performance in language models.
• A Stabilized Path-Space Approach to Diffusion-Based Posterior Sampling — A stabilized path-space framework improves diffusion posterior sampling by connecting it to stochastic optimal control and removing bias.
• LESS Is More: Mutual-Stability Sampling for Diffusion Language Models — Adaptive sampling based on token stability significantly enhances diffusion LLM inference efficiency.
• Nutpie: state-of-the-art mass matrix adaptation for HMC — The new "Nutpie sampler" significantly outperforms Stan's and PyMC's samplers, typically being twice as fast, primarily due to superior mass-matrix adaptation within the Hamiltonian Monte Carlo (HMC) framework.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.