SLPO: Scaling Latent Reasoning via a Surrogate Policy

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

Surrogate Latent Policy Optimization (SLPO), introduced on 2026-07-20 by researchers from The Hong Kong Polytechnic University and Sichuan University, addresses the computational cost of explicit Chain-of-Thought (CoT) reasoning and the imitation-bound nature of latent reasoners. Explicit CoT decodes every intermediate step as a language token, limiting scalability, while latent reasoning uses continuous vectors but lacks tractable per-step likelihoods and adaptive stopping interfaces for outcome-reward reinforcement learning (RL). SLPO overcomes these bottlenecks by defining an empirical surrogate policy density for trajectory-level credit assignment and equipping latent reasoners with a correctness-supervised stopping head. This head is refined by outcome-reward optimization into a variable-horizon policy, enabling adaptive computation. Empirically, SLPO improves Pass@k under parallel sampling and allocates longer latent computation to harder instances, demonstrated on models like Llama3.2-1B and Llama3.2-3B.

Key takeaway

For AI Scientists and Machine Learning Engineers developing advanced reasoning models, SLPO offers a critical path to move beyond imitation learning in latent reasoning. You should consider integrating SLPO's surrogate policy density and adaptive stopping head to enable outcome-reward optimization, allowing your models to dynamically allocate computational resources and achieve superior Pass@k performance on complex tasks. This approach can significantly enhance the scalability and efficiency of your latent reasoning systems.

Key insights

SLPO enables outcome-reward reinforcement learning for autoregressive latent reasoners by addressing key limitations.

Principles

• Latent reasoning lacks tractable per-step likelihoods for policy gradients.
• Fixed thinking budgets prevent adaptive computation in latent reasoners.
• Surrogate policy density can enable credit assignment in hidden space.

Method

SLPO defines an empirical surrogate policy density in hidden space for credit assignment and uses a correctness-supervised stopping head, refined by outcome-reward optimization, for variable-horizon adaptive computation.

In practice

• Apply SLPO to improve Pass@k in latent reasoning models.
• Enable adaptive allocation of latent computation based on task difficulty.

Topics

• Latent Reasoning
• Reinforcement Learning
• Chain-of-Thought
• Surrogate Policy Optimization
• Test-Time Scaling
• Adaptive Computation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• SLPO: Scaling Latent Reasoning via a Surrogate Policy — SLPO enables outcome-reward reinforcement learning for latent reasoners, overcoming their imitation-bound limitations.
• Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning — Strategy-Guided Policy Optimization distills reasoning by transferring reusable problem-solving strategies, not just specific solution steps.
• Structured Role-Aware Policy Optimization for Multimodal Reasoning — SRPO improves LVLM multimodal reasoning by assigning role-aware, token-level credit for visual evidence and reasoning.
• The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoning — Latent reasoning's core function is parameter shaping during training, not an active inference-time scratchpad.
• Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization — ISPO uses dense intrinsic signals to overcome common failure modes in RLVR for LLM reasoning tasks.
• Decompose, Look, and Reason: Reinforced Latent Reasoning for VLMs — DLR enhances VLM visual reasoning by dynamically decomposing queries and extracting premise-conditioned continuous visual latents.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.