Build On-Device AI Companions with the NVIDIA ACE Game Agent SDK and Unreal Engine 5 Plugins

2026-06-16 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Gaming & Interactive Media · Depth: Intermediate, medium

Summary

NVIDIA has introduced new tools for building on-device AI characters and gameplay within Unreal Engine 5, announced at Unreal Fest 2026. This includes the NVIDIA ACE Game Agent SDK, a C/C++ agentic framework for native in-game integration, optimized for small models and NVIDIA RTX hardware. Complementing this are new NVIDIA ACE Unreal Engine 5 plugins, offering local, runtime AI models for Automatic Speech Recognition (ASR) with nemo-conformer-ctc-120m, Small Language Models (SLM) with Qwen 3.5 4B, and Text-to-Speech (TTS) using Chatterbox Turbo 350M. These plugins support both Blueprint and C++ and provide an end-to-end AI NPC pipeline. Additionally, the DLSS 4.5 Unreal Engine plugin now supports Dynamic Multi Frame Generation and Multi Frame Generation 6X Mode. Other innovations include NVIDIA Kimodo for promptable human motion and SOKRISPYMEDIA's Chalk Warfare game utilizing local AI for weapon integration.

Key takeaway

For game developers building interactive NPCs in Unreal Engine 5, NVIDIA's new ACE Game Agent SDK and plugins offer a robust, on-device solution. You can integrate real-time speech recognition, dynamic dialogue, and expressive voice synthesis directly into your projects, bypassing cloud latency and costs. Utilize the SDK's APIs for complex agent behaviors or precise inference control. Explore the DLSS 4.5 plugin for enhanced rendering performance.

Key insights

NVIDIA's new SDK and plugins enable easier, high-performance on-device AI character development in Unreal Engine 5.

Principles

• On-device AI reduces latency and operational costs.
• Agentic frameworks drive multistep, tool-assisted reasoning.
• RAG architecture grounds AI responses in game data.

Method

The NVIDIA ACE Game Agent SDK provides Agent, Chat, and RAG APIs for dynamic NPC perception, reasoning, and action. The Unreal Engine 5 plugins integrate ASR, SLM, and TTS models locally.

In practice

• Use ACE SDK for stateful or stateless AI control.
• Implement RAG for context-aware NPC responses.
• Deploy local ASR, SLM, TTS models in Unreal Engine 5.

Topics

• NVIDIA ACE
• Unreal Engine 5
• On-Device AI
• Game AI
• DLSS 4.5
• Retrieval-Augmented Generation

Best for: NLP Engineer, AI Product Manager, AI Engineer, Software Engineer, Machine Learning Engineer

Related on AIssential

• What’s New for Game Developers in NVIDIA RTX: DLSS 4.5 for UE5 and Multilingual AI Characters — NVIDIA's RTX ecosystem updates enable more immersive games via multilingual AI characters and advanced rendering techniques.
• Tryll launches AI gaming engine alpha and secures $600K pre-seed funding — Tryll Engine enables on-device AI for games, reducing cloud reliance and simplifying integration for developers.
• Build Personal AI Agents on Windows PCs with New Tools from Microsoft and NVIDIA — NVIDIA and Microsoft are enabling secure, high-performance on-device AI agent development on Windows PCs.
• Nvidia Is Planning to Launch an Open-Source AI Agent Platform — Nvidia is entering the AI agent software space with an open-source platform.
• Nvidia CEO tries to explain why DLSS 5 isn’t just “AI slop” — Nvidia's DLSS 5 is an artist-guided generative AI tool for game enhancement, not post-processing, reflecting a broader strategy of co-design and ecosystem influence.
• NVIDIA Partners With Microsoft on Unified Stack for Agentic AI Deployment, From Windows Devices to Cloud to Local — NVIDIA and Microsoft are building a full-stack, secure, and scalable ecosystem for agentic AI, from edge devices to cloud.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.