Sample Complexity of Transfer Learning: An Optimal Transport Approach

2026-05-21 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, short

Summary

A new study rigorously analyzes the sample efficiency benefits of transfer learning, a crucial technique for complex AI models like large language models and generative AI, especially with low target task training data. Adopting an optimal transport perspective, the research establishes that for data dimensions d > 3, transfer learning exhibits a sample complexity of O(m^{-(α+1)/d}), where α denotes data distribution smoothness. This contrasts with O(m^{-p/d}) for direct learning, where p signifies optimal target model smoothness. This theoretical finding confirms superior sample efficiency for transfer learning, particularly when target tasks involve optimizing highly complex networks with non-smooth activation functions. Numerical demonstrations using image classification further illustrate significant performance improvements in data-hungry scenarios.

Key takeaway

For Machine Learning Engineers developing complex AI models with limited training data, you should prioritize transfer learning strategies. This approach offers superior sample efficiency, particularly when working with high-dimensional data (d > 3) and highly complex networks employing non-smooth activation functions. Implementing transfer learning can significantly enhance model performance in data-hungry scenarios, reducing the need for extensive new datasets and accelerating development cycles.

Key insights

Transfer learning significantly improves sample efficiency for complex models, especially with high data dimensions and non-smooth target models.

Principles

• Transfer learning offers O(m^{-(α+1)/d}) sample complexity for d > 3.
• Direct learning has O(m^{-p/d}) sample complexity.
• Optimal transport provides a framework for analyzing transfer learning.

Method

The study employs an optimal transport viewpoint to rigorously analyze sample complexity, comparing transfer learning against direct learning, and numerically demonstrates efficiency using image classification.

In practice

• Apply transfer learning for complex networks with limited data.
• Consider transfer learning for models using non-smooth activations.
• Use image classification as a benchmark for sample efficiency.

Topics

• Transfer Learning
• Sample Complexity
• Optimal Transport
• Machine Learning Theory
• Image Classification
• Generative AI

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization — Inverse entropic optimal transport provides a unified likelihood maximization framework for semi-supervised conditional distribution learning using mixed data.
• The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity — The γ-graph dimension characterizes learnability, and a median-based aggregation of three interpolating hypotheses is optimal.
• Your GFlowNet Secretly Learns an Optimal Transport Plan — GFlowNets can learn optimal transport plans by reducing their objective to a Kantorovich OT problem.
• Explaining Data Mixing Scaling Laws — A unified framework explains data mixing scaling laws through capacity competition and noise reduction.
• Transfer Learning in Machine Learning and Deep Learning — Transfer learning reuses pre-trained model knowledge to accelerate new task learning with less data.
• Geometric Dictionary Learning of Dynamical Systems with Optimal Transport — DOODL learns shared spectral manifolds of dynamical systems for robust, data-efficient operator estimation and system identification.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.