Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind

2026-05-20 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

Google DeepMind's Patrick Löber presented the "Any-to-Any" vision for building native multimodal agents using the Gemini API and AI Studio. This approach utilizes Gemini's native multimodal understanding, processing text, code, image, audio, and video inputs. While the main Gemini 3 model outputs text, specialized native generation models like Nano Banana 2 (for image) and a Gemini 2.5-based model (for speech) handle other modalities. The presentation detailed an agentic architecture where Gemini acts as a reasoning model, using function calls to orchestrate these specialized generators. Practical tips included using Gemini 3 flash for audio transcription (1 minute audio ≈ 1920 tokens, 1 million token limit allows >9 hours audio, 1 hour video), context caching for 90% cost savings on repeated queries, and the new Gemini 3.1 flash live model for real-time audio-to-audio interactions.

Key takeaway

For AI Engineers building multimodal applications, you should utilize the Gemini API's agentic architecture to combine understanding and generation. Configure Gemini as a reasoning model to orchestrate specialized tools like Nano Banana 2 for image creation or the text-to-speech model for audio summaries. This approach allows your agents to dynamically decide content creation, enhancing user interaction and reducing development complexity by abstracting multimodal workflows. Explore the Live API for real-time audio-to-audio conversational agents.

Key insights

Gemini enables "any-to-any" multimodal agents by combining native understanding with specialized generation models via function calls.

Principles

• Agentic loops orchestrate specialized models.
• Native generation models understand context.
• Context caching reduces repeated query costs.

Method

Use Gemini as a reasoning model with function calling to invoke specialized generation models (e.g., Nano Banana 2 for images, Gemini 2.5 for speech) based on input analysis.

In practice

• Transcribe audio with Gemini Flash for summaries.
• Use context caching for large, repeated file queries.
• Build agents that generate infographics and audio summaries.

Topics

• Multimodal AI
• Gemini API
• Agentic AI
• Function Calling
• Nano Banana
• Real-time AI

Best for: AI Engineer, Machine Learning Engineer, AI Student

Related on AIssential

• Generative AI News Rundown - A Deep Dive Into ChatGPT-4o, Gemini Upgrades and Intrigue & More - Voicebot Podcast Ep 381 — Multimodal AI models are rapidly advancing, integrating voice, vision, and reasoning for more natural and agentic user experiences.
• Gemini Interactions API Quick Start — The Gemini Interactions API unifies agentic application development with server-side state, tool orchestration, and multimodal capabilities.
• Build a Multimodal Agentic RAG App with Gemini Embedding 2 and Google ADK — Gemini Embedding 2 and Google ADK enable unified multimodal RAG with consistent citations across diverse data types.
• Kimi 2.6 Test | The Best Agentic Coding Open Model? | Coding, OCR, Image Understanding | 🔴 Live — Gemini K 2.6 is a large, multimodal open model excelling in agentic coding and visual design tasks.
• 9 demos of Gemini Omni and Gemini 3.5 in action — Gemini Omni and 3.5 Flash advance multimodal AI, enabling conversational video editing and intelligent agentic workflows.
• The Gen AI Power Behind Google's Latest Search Updates — Google's search engine is evolving into an intelligent agent, using Gen AI to anticipate intent and automate complex tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.