Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

2026-02-17 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Ferret-UI Lite is a compact, end-to-end GUI agent designed for on-device operation across mobile, web, and desktop platforms, developed by Zhen Yang et al. and published in February 2026. This 3-billion-parameter model was built using techniques optimized for small models, including curating diverse GUI data from real and synthetic sources, enhancing inference-time performance via chain-of-thought reasoning and visual tool-use, and applying reinforcement learning with custom rewards. Ferret-UI Lite demonstrates competitive performance among small-scale GUI agents, achieving 91.6% on ScreenSpot-V2, 53.3% on ScreenSpot-Pro, and 61.2% on OSWorld-G for GUI grounding. For GUI navigation, it reached success rates of 28.0% on AndroidWorld and 19.8% on OSWorld. This work shares methods and lessons from developing compact, on-device GUI agents.

Key takeaway

For AI Scientists developing on-device GUI agents, Ferret-UI Lite demonstrates that competitive performance is achievable with compact models. You should consider integrating diverse data mixtures, chain-of-thought reasoning, and visual tool-use, alongside reinforcement learning, to optimize your agent's efficiency and accuracy on resource-constrained platforms.

Key insights

Ferret-UI Lite is a compact, end-to-end GUI agent optimized for on-device performance across diverse platforms.

Principles

• Diverse data improves GUI agent performance.
• Chain-of-thought enhances inference for small models.
• Reinforcement learning refines agent behavior.

Method

The Ferret-UI Lite agent was built by curating diverse GUI data, strengthening inference with chain-of-thought and visual tool-use, and applying reinforcement learning with designed rewards.

In practice

• Use mixed real and synthetic GUI data.
• Implement visual tool-use for GUI tasks.
• Apply RL for on-device agent optimization.

Topics

• On-Device GUI Agents
• User Interface Understanding
• Reinforcement Learning
• Chain-of-Thought Reasoning
• Multimodal LLMs

Best for: AI Scientist, Research Scientist, AI Researcher, AI Engineer, Machine Learning Engineer

Related on AIssential

• GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL — Tailored training and data curation significantly boost GUI agent performance on complex navigation tasks.
• PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation — PerceptUI uses LLM agents trained on human decisions and self-correction to provide persona-conditioned, realistic UI/UX evaluations.
• Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale — Efficient, scalable agentic intelligence at trillion-parameter scale is achieved through co-design and specialized architectural and training innovations.
• Theory of Mind in Action: The Instruction Inference Task in Dynamic Human-Agent Collaboration — Few-shot chain-of-thought prompting significantly enhances LLM Theory of Mind for collaborative task interpretation.
• 4 Layer "AI Harness" For LLMs (+54%). Really? — The "AI Harness" improves LLM agent performance by addressing interface mismatches and syntax issues, not by modifying the frozen model.
• From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google — Fine-tuning tiny LLMs with synthetic data dramatically enhances on-device agent performance and reliability for specific tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.