SODA: Semi On-Policy Black-Box Distillation for Large Language Models

2023-03-30 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

SODA (Semi On-policy Distillation with Alignment) is a novel, highly efficient framework for black-box knowledge distillation of large language models. It addresses the trade-off between simple off-policy methods like SeqKD, which struggle with student errors, and fully on-policy methods like GAD, which incur high computational overhead and instability. SODA constructs an effective contrastive signal by pairing a frontier teacher model's superior responses with a one-time static snapshot of a smaller student model's inferior zero-shot outputs. This approach, utilizing Direct Preference Optimization, eliminates the need for costly dynamic rollouts or fragile adversarial training. Evaluations across Qwen2.5 and Llama-3 models (3B-14B) with GPT-5-Chat as teacher show SODA matches or outperforms leading methods on 15 out of 16 benchmarks, while training 10x faster and using 27% less peak GPU memory.

Key takeaway

For Machine Learning Engineers tasked with distilling large language models for efficient deployment, SODA offers a compelling alternative to complex on-policy methods. You can achieve top-tier distillation quality, matching or exceeding GAD, while benefiting from 10x faster training and 27% less peak GPU memory. This approach eliminates the instability of adversarial training, making your distillation pipeline more robust and resource-friendly. Consider integrating SODA to efficiently produce capable small models from proprietary teachers.

Key insights

A static snapshot of a student's own inferior responses provides an efficient, targeted contrastive signal for black-box LLM distillation.

Principles

• Semi on-policy distillation can match fully on-policy quality.
• Student's own prior outputs are optimal negative examples.
• Targeted error correction improves model representation diversity.

Method

Generate base student responses offline, perform a brief supervised warmup, then apply DPO using teacher responses as preferred and base student responses as rejected.

In practice

• Generate q0 responses once for DPO negative examples.
• Expect 10x faster training, 27% less GPU memory.
• Eliminate adversarial training instability.

Topics

• Black-box Knowledge Distillation
• Large Language Models
• Direct Preference Optimization
• Semi On-policy Learning
• Model Efficiency
• LLM Alignment

Code references

• tatsu-lab/stanford_alpaca

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Draft-OPD: On-Policy Distillation for Speculative Draft Models — Draft-OPD improves speculative draft models by learning from their own policy's errors during target-assisted rollouts.
• Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMs — Answer-conditioned CoT distillation significantly improves small VLM performance for few-shot industrial vision by ensuring correct reasoning.
• EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models — EasyOPD unifies fragmented on-policy distillation methods for LLMs through a shared verl-based framework, improving reproducibility and extensibility.
• Distribution Corrected Offline Data Distillation for Large Language Models — Offline distillation can achieve high quality and efficiency by correcting teacher-student distribution drift.
• Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition — Chain-of-Thought distillation significantly improves compact models' mathematical reasoning, with response length being a critical factor.
• A Survey of On-Policy Distillation for Large Language Models — OPD overcomes off-policy distillation's exposure bias by enabling LLMs to learn from self-generated trajectories with teacher feedback.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.