Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Internet of Things (IoT) & Connected Devices · Depth: Expert, quick

Summary

A study investigated distilling an 8B reasoning teacher model, deepseek-r1:8b, into a 0.6B student model, Qwen3-0.6B, for on-device structured text enrichment. The process involved mapping news articles to JSON objects containing a short summary and five categorical labels. The student model achieved a processing time of 0.8 seconds per article, significantly faster than the teacher's 39 seconds. It recovered 58% of the base-to-teacher gap in summary quality, surpassing constrained decoding by +16.8 points and few-shot prompting by +4.9 points. This summary gain was attributed to the teacher's reasoning capabilities rather than its scale. The research found that different teachers impart distinct capabilities: the reasoning teacher transferred writing quality, while a managed pipeline transferred label diversity. A same-size instruction teacher's student demonstrated better grounding on short articles, fabricating less (74 versus 55 faithful) compared to the reasoning-lineage student. The findings suggest a per-field routing map is necessary for optimal on-device enrichment.

Key takeaway

For Machine Learning Engineers deploying sub-1B models for on-device structured text enrichment, your teacher model selection is paramount. If your primary goal is high-quality summarization, prioritize distilling from a reasoning-capable teacher. Conversely, for tasks requiring diverse categorical labeling, consider teachers like managed pipelines. You should implement a per-field routing strategy to optimize output quality across different enrichment tasks, acknowledging that no single distilled model will excel universally.

Key insights

Distilling large reasoning models into small on-device models significantly reduces latency while retaining substantial quality, but teacher choice dictates specific capability transfer.

Principles

• Reasoning teachers transfer writing quality effectively.
• Teacher scale alone does not guarantee summary quality gain.
• Different teachers excel at transferring distinct capabilities.

Method

An 8B reasoning teacher (deepseek-r1:8b) was distilled into a 0.6B student (Qwen3-0.6B) using QLoRA across three seeds, evaluated by a three-judge panel against baselines.

In practice

• Use reasoning teachers for summary quality.
• Consider managed pipelines for label diversity.
• Route tasks to specific models based on required output quality.

Topics

• On-device AI
• Model Distillation
• Structured Text Enrichment
• Reasoning Models
• QLoRA
• Large Language Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment — Distilling reasoning teachers into small models significantly boosts summary quality and speed for structured text enrichment.
• Model Distillation Guide: Compressing LLMs for Edge Efficiency — Model distillation compresses large language models into smaller, efficient versions using a teacher-student framework.
• Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation — Pedagogically-inspired knowledge distillation systematically improves student model performance and efficiency.
• The Sequence Knowledge #882: A New Series About Distillation — Distillation is becoming central to AI development, addressing the practical limitations of increasingly large and expensive models.
• What properties of reasoning supervision are associated with improved downstream model quality? — Intrinsic data metrics can reliably predict reasoning dataset utility before model training, with scale-dependent predictors.
• When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization — Standard knowledge distillation often harms student performance; selective, reliability-aware methods significantly improve low-resource summarization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.