Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

2026-07-01 · Source: Latent Space: The AI Engineer Podcast · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, extended

Summary

Modal, a cloud platform, recently secured a \$355M Series C, signaling its leadership in evolving AI infrastructure for "agent experience." The company's CTO, Akshat Bubna, explains that traditional cloud environments, including Kubernetes, are ill-suited for the bursty, compute-intensive demands of AI workloads and autonomous agents. Modal addresses this by offering a specialized stack featuring serverless functions, decorator-based infrastructure, elastic inference for custom models (across audio, video, robotics, and computational biology), GPU snapshotting, and advanced LLM inference techniques like DeFlash and speculative decoding. The platform also provides Auto Endpoints, sandboxes (critical for RL rollouts that can require 100,000 instances), persistent storage, and multi-node training. Modal leverages a "supercloud" strategy, aggregating capacity across 17 cloud providers to ensure high elasticity and reliability for AI applications.

Key takeaway

For AI Engineers deploying agentic applications or custom models, traditional cloud infrastructure will likely prove inefficient for bursty, compute-heavy workloads. You should evaluate specialized platforms like Modal that offer primitives for agent experience, including elastic inference, GPU snapshotting, and sandboxes. This approach ensures faster iteration, better observability, and robust guardrails for production-grade AI systems, optimizing both performance and cost.

Key insights

AI agents necessitate a programmatic, elastic infrastructure with rapid feedback loops, diverging from traditional human-centric cloud development.

Principles

• Traditional cloud infrastructure is ill-suited for bursty, compute-heavy AI workloads.
• Agent experience demands infrastructure co-located with code for rapid iteration.
• Observability is paramount when agents generate and operate code.

Method

Modal builds AI cloud primitives from scratch, including serverless functions, decorator-based infra, GPU snapshotting, and a 17-cloud capacity pool. It optimizes LLM inference via speculative decoding and Auto Endpoints.

In practice

• Apply GPU snapshotting for faster cold starts in bursty inference.
• Deploy networked sandboxes with sidecars for complex agent environments.
• Leverage private IPv6 overlay networks for secure inter-container communication.

Topics

• AI Infrastructure
• Agent Experience
• Elastic Inference
• GPU Sandboxes
• Speculative Decoding
• Multi-node Training
• Cloud Capacity Management

Best for: Investor, CTO, VP of Engineering/Data, MLOps Engineer, AI Engineer, AI Architect

Related on AIssential

• Build Small with Modal — Modal offers purpose-built, developer-friendly infrastructure for scaling data and compute-intensive AI workloads with fast GPU orchestration and sandboxing.
• [AINews] New AI Infra unicorns: Exa, Modal, TurboPuffer — The AI ecosystem is rapidly maturing, marked by significant infrastructure funding, model advancements, and agent-centric product development.
• Daytona raises $24M Series A to build agent-native compute infrastructure — Agent-native compute infrastructure is crucial for scaling stateful, long-running AI agent workloads beyond traditional cloud models.
• Forward Future Live | 02.06.26 | Guests from Modular, Emergence Capital, & Axiom — AI models are rapidly advancing in context, agentic capabilities, and self-improvement, intensifying competition and reshaping industry landscapes.
• Architecting Modern AI Systems — Local hosting of open-source models offers cost control, data sovereignty, and predictable performance over closed-source APIs.
• The Agent Stack - Part 2: Foundation Infrastructure, Models, and Inference — Agent system robustness depends on understanding and explicitly managing underlying infrastructure and model engine semantics.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Latent Space: The AI Engineer Podcast.