From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

2026-05-20 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Internet of Things (IoT) & Connected Devices · Depth: Advanced, long

Summary

Google's AI Edge stack, featuring the Lighter TLM runtime, enables on-device AI processing for latency, privacy, and offline use. It supports two main approaches: system-level GenAI like Gemini Nano via AI Core for optimized, preloaded models (e.g., Gemma 4 E2B/E4B), and App GenAI using Lighter TLM for highly customized applications with Tiny LLMs (under 1 billion parameters). The Google AI Edge Gallery app showcases agent skills built on Gemma 4, demonstrating how models can dynamically load and execute JavaScript-based tools. A key finding highlights that fine-tuning Function Gemma, a 270-million-parameter model, with synthetic data dramatically improved function calling success from 46% to over 90% for eight out of ten functions. This robust fine-tuning workflow allows developers to deploy highly reliable, small models, as exemplified by the Eloquent transcription app, which chains multiple Gemma 3-based Tiny LLMs.

Key takeaway

For mobile app developers or AI engineers building on-device agents, consider Google's AI Edge stack and Lighter TLM for highly customized solutions. While system-level GenAI offers convenience, fine-tuning Tiny LLMs like Function Gemma with synthetic data can elevate task-specific performance from 46% to over 90%. This approach, though requiring more effort, enables robust, privacy-preserving, and low-latency AI directly within your application, expanding capabilities beyond pre-installed system models. Explore the Function Gemma fine-tuning lab to achieve reliable function calling.

Key insights

Fine-tuning tiny LLMs with synthetic data dramatically enhances on-device agent performance and reliability for specific tasks.

Principles

• On-device AI processing provides benefits in latency, privacy, and offline functionality.
• Fine-tuning is essential for achieving robust, high-accuracy performance with small, task-focused LLMs.
• Agent skills can be dynamically integrated into apps using simple prompts and JavaScript-based tool calls.

Method

Export models from Transformers using lighter.torch, then fine-tune Tiny LLMs with synthetically generated datasets, and deploy them via the Lighter TLM runtime.

In practice

• Develop custom agent skills for mobile apps using Gemma 4 and integrate JavaScript for UI interactions.
• Implement robust offline transcription services by chaining multiple fine-tuned Tiny LLMs.
• Utilize the Function Gemma fine-tuning lab to improve function calling accuracy for on-device agents.

Topics

• On-Device AI
• Tiny LLMs
• Fine-tuning
• Agentic AI
• Google AI Edge
• Lighter TLM
• Gemma

Best for: NLP Engineer, AI Architect, AI Product Manager, AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google — Edge AI enables low-latency, private, and cost-effective on-device model deployment, leveraging tiny LLMs and agent skills.
• Is Fine-Tuning Still Valuable? — Effective LLM fine-tuning requires robust evaluation systems and iterative data curation, especially for domain-specific tasks.
• This AI Paper Introduces TinyLoRA, A 13-Parameter Fine-Tuning Method That Reaches 91.8 Percent GSM8K on Qwen2.5-7B — TinyLoRA enables high LLM performance with only 13 parameters by leveraging reinforcement learning.
• Gemma 4 Fine-Tuning on Single GPU | Training Gemma 4 With Unsloth on Custom Dataset (🔴 Live) — Fine-tuning LLMs may offer better accuracy and lower GPU RAM than prompting alone.
• Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automation — Multi-agent LLMs can automate testbench generation, creating high-quality data for efficient Verilog code generation.
• The ‘brownie recipe problem’: why LLMs must have fine-grained context to deliver real-time results — LLMs require fine-grained, real-time context and modular architectures to deliver effective, low-latency assistance in dynamic systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.