Capturing token IDs during agentic interactions for better reinforcement learning

2026-07-09 · Source: Amazon Science homepage · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, extended

Summary

Amazon AGI Labs has released Turnstile, an open-source Rust proxy designed to improve reinforcement learning (RL) for agentic language models. Available since July 9, 2026, Turnstile addresses the critical issue of "retokenization drift" by capturing exact token IDs, per-token log probabilities, and loss masks directly between the agent harness and the model backend. This ensures RL trainers receive an unambiguously correct record of what the model actually generated, unlike unreliable text transcripts. Turnstile supports prefix-aware multiturn trajectories, optional mixture-of-experts (MoE) routing capture, and multimodal rollouts, including processed image features. Validations demonstrated steady RL improvements for a Qwen3-1.7B coding agent on MBPP and a Qwen3-VL-8B multimodal agent on OSWorld tasks, without modifying existing harnesses.

Key takeaway

For Machine Learning Engineers developing agentic LLMs, if you are struggling with degraded reinforcement learning signals due to token-level inconsistencies, integrate Turnstile into your training pipeline. This Rust proxy captures precise token IDs, log probabilities, and loss masks directly, ensuring your RL trainer optimizes against the exact context the model experienced. You can maintain your existing agent harnesses, avoiding complex modifications, and achieve more reliable and effective model improvements.

Key insights

Accurate token-level data, not text transcripts, is crucial for effective reinforcement learning in agentic LLMs.

Principles

• RL training demands exact token-level history.
• Text transcripts introduce "retokenization drift".
• Capture token data at generation for fidelity.

Method

Turnstile proxies OpenAI Chat Completions API calls, recording token IDs, log probabilities, and loss masks at generation. It exports framework-neutral trajectories for RL training.

In practice

• Point your agent harness to Turnstile's address.
• Enable MoE capture for mixture-of-experts models.
• Utilize multimodal support for vision-language models.

Topics

• Reinforcement Learning
• Agentic LLMs
• Tokenization
• Data Capture
• Multimodal AI
• Mixture-of-Experts

Code references

• amazon-agi-labs/turnstile
• xlang-ai/OSWorld
• All-Hands-AI/OpenHands
• THUDM/slime
• PrimeIntellect-ai/renderers

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• not much happened today — Agentic AI development is converging on integrated stacks, robust infrastructure, and careful token economics for performance and cost.
• Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning — ICT stabilizes LLM reasoning by focusing on token-level distributional deviations, preventing entropy collapse and explosion.
• Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems — Token-optimized formats like TRON can significantly reduce LLM token consumption in agentic systems with minimal accuracy loss.
• NVIDIA Releases Polar, a Token-Faithful Rollout Framework for GRPO Training Across Codex, Claude Code, and Qwen Code — Polar simplifies RL system integration by abstracting the agent harness via a model API proxy.
• Leyline: KV Cache Directives for Agentic Inference — Leyline enables efficient, policy-driven KV cache editing for agentic LLMs, avoiding costly full re-prefills.
• Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate — Early-token confidence from LLM decoding reliably predicts reasoning quality in multi-agent systems.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Amazon Science homepage.