How Together AI Uses NVIDIA's Full Stack to Deliver AI Responses in Under 100ms

2026-06-30 · Source: NVIDIA · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, quick

Summary

Together AI, an AI cloud and inference provider, focuses its research on optimizing models for NVIDIA GPUs for ultra-low latency responses. The company developed the "megakernel," which integrates an entire model into a single kernel. This enables real-time voice agents to receive 64 words within 100 milliseconds. Another project, Together ATLAS, dynamically adapts speculative decoders to user traffic for enhanced speed. Together AI heavily relies on the NVIDIA ecosystem, utilizing libraries such as CUDA and CUTLASS. They also use NVIDIA Dynamo and NVIDIA TensorRT to build deep AI solutions. These technologies support demanding applications like cogeneration. This involves long-context, multi-turn interactions requiring extremely low latency, as demonstrated by customers like Cursor. Together AI anticipates further advancements with future NVIDIA platforms like Blackwell and Vera Rubin.

Key takeaway

For AI Engineers building real-time applications, Together AI's approach demonstrates how deep GPU optimization can achieve sub-100ms response times. You should investigate custom kernel development, like the megakernel, and adaptive decoding techniques to meet stringent latency requirements. Consider employing NVIDIA's full stack, including CUDA, CUTLASS, and TensorRT, to accelerate long-context, multi-turn AI applications. This strategy is crucial for delivering highly responsive user experiences in demanding scenarios.

Key insights

Together AI optimizes AI inference on NVIDIA GPUs using custom kernels and adaptive decoders for sub-100ms responses.

Principles

• Deep research optimizes GPU model performance.
• Single-kernel models reduce inference latency.
• Adaptive decoders improve speed with user traffic.

Method

Together AI uses megakernels to consolidate models into single GPU kernels and Together ATLAS to adapt speculative decoders to real-time user traffic.

In practice

• Achieve 64 words in 100ms for voice agents.
• Support long-context, multi-turn cogeneration.
• Utilize CUDA, CUTLASS, TensorRT for optimization.

Topics

• AI Inference Optimization
• NVIDIA GPUs
• Low-latency AI
• Megakernel
• Speculative Decoding
• NVIDIA Ecosystem

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Speeding up GPU kernels by 38% with a multi-agent system — Multi-agent systems can autonomously optimize complex software, achieving significant performance gains in weeks.
• How NVIDIA GPUs Powered the AI Revolution — NVIDIA GPUs and CUDA enabled the AI revolution by providing parallel processing power for deep learning.
• From RTX to Spark: NVIDIA Accelerates Gemma 4 for Local Agentic AI — Gemma 4 models, optimized for NVIDIA GPUs, enable efficient on-device AI with multimodal and multilingual capabilities.
• Inside the Together AI kernels team — GPU kernel optimization, exemplified by FlashAttention, offers substantial performance gains for AI workloads.
• Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support — TensorRT 11.0 scales generative AI inference across multiple GPUs using NCCL and parallelism strategies.
• Nvidia Already Won Training. The Real Fight Is Inference — AI inference hardware is shifting from general-purpose GPUs to specialized architectures optimizing for latency and memory access.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA.