Streaming Tokens and Tools: Multi-Turn Agentic Harness Support in NVIDIA Dynamo

2026-05-08 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, long

Summary

NVIDIA Dynamo has been enhanced to improve correctness, user experience, and performance for agentic inference workflows, building on prior work that optimized serving architecture. This update focuses on hardening parser and API coverage, improving streaming behavior, and extracting parser layers into reusable crates. Key improvements include `--strip-anthropic-preamble` to restore KV cache reuse by removing session-specific billing headers, which reduced time to first token (TTFT) by approximately 5x on a 52K-token prompt. Dynamo now correctly handles interleaved reasoning and tool calls, preventing reordering and aggressive dropping of reasoning, which previously caused a 1.9x TTFT increase. Additionally, `--enable-streaming-tool-dispatch` allows immediate tool execution and streaming of tool-call dispatch events, enhancing responsiveness. Fidelity with Anthropic Messages API (for Claude Code, OpenClaw) and OpenAI Responses API (for Codex) has also been improved, addressing issues like model metadata handling and correct `input_tokens` reporting, which are crucial for agent behavior and context management.

Key takeaway

For AI Architects and NLP Engineers deploying agentic models, ensuring prompt stability and accurate reasoning/tool parsing is paramount for performance and correctness. Configure NVIDIA Dynamo with `--strip-anthropic-preamble` to significantly reduce TTFT by enabling KV cache reuse, and utilize `--enable-streaming-tool-dispatch` to enhance user experience by allowing immediate tool execution. Pay close attention to model-specific reasoning replay policies and API fidelity to avoid silent malformation or dropped context, which can degrade agent performance and lead to incorrect behavior.

Key insights

Optimizing agentic inference requires precise prompt stability, correct reasoning/tool parsing, and efficient streaming.

Principles

• Prompt stability is critical for KV cache reuse.
• Reasoning replay is model- and turn-dependent.
• Streaming tool dispatch improves responsiveness.

Method

Dynamo uses specific flags like `--strip-anthropic-preamble` and `--enable-streaming-tool-dispatch`, alongside dedicated reasoning and tool-call parsers, to manage prompt stability, reasoning replay, and streaming behavior for agentic workflows.

In practice

• Strip unstable headers to improve KV cache hits.
• Ensure parsers correctly handle interleaved reasoning and tool calls.
• Enable streaming tool dispatch for faster execution.

Topics

• NVIDIA Dynamo
• Agentic Workflows
• KV Cache Optimization
• Streaming Tool Dispatch
• Reasoning Parsing

Code references

• ai-dynamo/dynamo
• karpathy/autoresearch

Best for: NLP Engineer, AI Architect, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Nvidia’s new technique cuts LLM reasoning costs by 8x without losing accuracy — Dynamic Memory Sparsification reduces LLM memory costs by intelligently compressing the KV cache without accuracy loss.
• NVIDIA AI Introduces PivotRL: A New AI Framework Achieving High Agentic Accuracy With 4x Fewer Rollout Turns Efficiently — PivotRL enhances agentic accuracy and efficiency by selectively applying RL to SFT trajectories.
• NVIDIA Releases Nemotron-Cascade 2: An Open 30B MoE with 3B Active Parameters, Delivering Better Reasoning and Strong Agentic Capabilities — NVIDIA's Nemotron-Cascade 2 MoE model excels in reasoning and agentic tasks via Cascade RL and MOPD.
• NVIDIA Achieves Leading Agentic Coding Performance on First Agentic AI Benchmark — AA-AgentPerf defines the first standard for measuring AI agentic coding performance, revealing significant hardware efficiency gains.
• Build with DeepSeek V4 Using NVIDIA Blackwell and GPU-Accelerated Endpoints — DeepSeek V4 models optimize long-context inference via a hybrid attention mechanism, significantly reducing FLOPs and KV cache memory.
• UC San Diego Lab Advances Generative AI Research With NVIDIA DGX B200 System — Disaggregated inference, optimizing "goodput" by separating prefill and decode, enhances LLM serving efficiency and user experience.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.