Control an Android Phone with Gemini 3.5 Flash Computer Use

2026-06-25 · Source: philschmid.de - RSS feed · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Intermediate, medium

Summary

Google's Gemini 3.5 Flash model now features "Computer Use," enabling programmatic control of Android devices via a screenshot-action loop. The model analyzes a device screenshot and a goal, then returns structured function calls like `click(y=300, x=500)`. These actions are executed on the device using ADB, a new screenshot is captured, and the process repeats until the task is complete. This guide specifically details controlling an Android emulator using the `mobile` environment and the Python SDK, providing pseudocode and a full `agent.py` script. Supported actions include `open_app`, `click`, `type`, `long_press`, `drag_and_drop`, and `press_key`, with coordinates normalized to a 0-999 grid. Setup involves a `setup_emulator.sh` script and `pip install google-genai`. The system also supports connecting to remote physical or cloud-hosted Android devices by passing a `device_id`.

Key takeaway

For AI Engineers or ML Engineers developing mobile automation solutions, Gemini 3.5 Flash's "Computer Use" provides a powerful, visual-driven approach. You can utilize its ability to interpret screenshots and generate precise actions to automate complex Android tasks, from app navigation to system settings. Consider integrating this Python SDK-based framework to build intelligent agents that interact with mobile UIs more dynamically and robustly than traditional scripting.

Key insights

Gemini 3.5 Flash's "Computer Use" enables AI agents to control mobile devices by interpreting screenshots and executing structured actions.

Principles

• AI control via visual feedback loop.
• Normalized coordinates simplify device interaction.
• Model-agnostic action output for cross-platform.

Method

The method involves an agent loop: Gemini 3.5 Flash receives a screenshot and goal, outputs function calls, which are executed via ADB, then a new screenshot is captured and fed back.

In practice

• Automate Android tasks like settings changes.
• Develop cross-platform mobile automation agents.
• Integrate AI for complex device interactions.

Topics

• Gemini 3.5 Flash
• Computer Use API
• Android Automation
• Mobile AI Agents
• ADB Control
• Function Calling

Code references

• google-gemini/gemini-android-computer-use-quickstart
• danielpaulus/go-ios

Best for: AI Engineer, Machine Learning Engineer, AI Student

Related on AIssential

• Google bakes computer control directly into Gemini 3.5 Flash, letting the model see and operate your screen — Gemini 3.5 Flash now directly controls computers, enabling autonomous agent development.
• Decoding Gemini Intelligence: How automated workflows will change Android - Business Standard — Gemini Intelligence transforms Android into a proactive, context-aware AI system capable of multi-app task automation.
• Introducing computer use in Gemini 3.5 Flash — Gemini 3.5 Flash now natively supports "computer use" for building cross-platform automation agents with enhanced safety features.
• Gemini 3.5: frontier intelligence with action — Gemini 3.5 Flash delivers frontier agentic and coding performance at high speed and reduced cost.
• Introducing computer use in Gemini 3.5 Flash — Gemini 3.5 Flash now natively supports "computer use" for building advanced, cross-platform automation agents with enhanced safety features.
• Google launches Gemini 3.5 Flash as its default AI model — Google's Gemini 3.5 Flash offers high-speed, cost-effective AI for agentic tasks, becoming the default model.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by philschmid.de - RSS feed.