NVIDIA's AI Engineers: Agent Inference at Planetary Scale and "Speed of Light" — Nader Khalil (Brev), Kyle Kranen (Dynamo)

2026-03-06 · Source: Latent Space: The AI Engineer Podcast · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, extended

Summary

NVIDIA GTC will feature sessions with Kyle Kranen, an architect of NVIDIA Dynamo, and Nader Khalil, from NVIDIA Brev, discussing the company's rapid growth and strategic investments in AI. Dynamo is presented as a data center scale inference engine that optimizes serving by leveraging techniques like prefill/decode disaggregation, scheduling, and Kubernetes-based orchestration, addressing critical tradeoffs in cost, latency, and quality. Khalil highlights Brev's role in simplifying GPU access and NVIDIA's broader focus on developer experience, including the "SOL" (Speed of Light) urgency concept. The discussion also delves into agent security models, the "system as model" paradigm for complex AI workflows, and the importance of model/hardware co-design for managing long context lengths and future AI advancements. The team also explores the potential for "unhobblers"—scientific discoveries that drastically improve scaling—to overcome current limitations in context length.

Key takeaway

NVIDIA Dynamo optimizes datacenter-scale LLM inference by disaggregating prefill and decode phases, leveraging Kubernetes for specialized scaling to manage cost, quality, and latency tradeoffs. This enables efficient resource allocation for compute-bound prefill and memory-bound decode, exemplified by Kimi-2's 128K context fitting in just 8GB VRAM through hardware/model co-design. AI/ML professionals can utilize Dynamo for high-performance LLM and agent deployments, complemented by NVIDIA Brev's simplified GPU access and secure agent permission models.

Topics

• AI Inference Optimization
• NVIDIA Dynamo
• Agent-based AI Systems
• GPU Developer Tools
• Model-Hardware Co-design

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Machine Learning Engineer, MLOps Engineer, AI Architect

Related on AIssential

• Nvidia launches Dynamo 1.0 AI inference operating system — Nvidia has launched Dynamo 1.0, an open-source "operating system" for large-scale AI inference, now in production and available to developers globally.
• Scaling Agentic Inference Across Heterogeneous Compute [Zain Asgar] - 757 — Heterogeneous inference orchestration across diverse hardware significantly improves AI workload efficiency and cost-per-token.
• The Nvidia AI PC, Project Solara, Microsoft AI — The AI landscape is shifting from local inference PCs to cloud-centric agent platforms and customizable enterprise models.
• AWS and NVIDIA deepen strategic collaboration to accelerate AI from pilot to production — AWS and NVIDIA are deepening their partnership to provide integrated, scalable, and secure AI infrastructure and services.
• Unlocking High-Performance Inference for DeepSeek with NVFP4 on NVIDIA Blackwell — Blackwell GPUs with NVFP4 quantization and TensorRT LLM dramatically boost MoE model inference performance.
• Maximizing GPU Utilization with NVIDIA Run:ai and NVIDIA NIM — Intelligent orchestration with NVIDIA NIM and Run:ai significantly boosts GPU utilization and inference performance for LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Latent Space: The AI Engineer Podcast.