I Fine-Tuned DistilBERT Twice. It Humbled Me Both Times.

2026-05-22 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

An analysis comparing two fine-tuning runs of DistilBERT reveals that data characteristics, not model architecture or training setup, dictate performance and interpretation. The same DistilBERT model, trained for three epochs, achieved a weighted F1 of 0.9070 on the SST-2 dataset (67K binary sentiment movie reviews) but only 0.7261 on 91K real-world consumer complaints across eight categories. Both runs exhibited overfitting from epoch 2, with validation loss rising. However, the SST-2 task had clean, consistent signals, while the consumer complaints task showed conflicting metrics (rising validation loss but improving F1), indicating miscalibration. The significant F1 score difference is attributed to the consumer complaints dataset's complexity, featuring eight imbalanced classes (29.1% top, 5.3% bottom) and domain-specific vocabulary, which a general-purpose pretrained model struggles with without further intervention like domain-adapted pretraining or class-weighted loss.

Key takeaway

For Machine Learning Engineers fine-tuning pre-trained models for classification, understand that your dataset's characteristics profoundly impact model performance and metric interpretation. If you observe conflicting training signals or lower-than-expected F1 scores, consider the data's class balance and domain specificity. You should prioritize metrics like weighted F1 for imbalanced data and explore domain-adapted models or class-weighted loss to overcome dataset ceilings, rather than solely adjusting training parameters.

Key insights

Data characteristics fundamentally determine fine-tuning performance and metric interpretation, not just the model.

Principles

• Metric choice depends on dataset characteristics.
• High scores don't always mean easy problems.
• Dataset complexity sets a real performance ceiling.

Method

Compare identical model setups on diverse datasets to reveal data-specific performance nuances and interpret conflicting training signals.

In practice

• Use weighted F1 for imbalanced datasets.
• Consider domain-adapted models for specific vocabularies.
• Implement class-weighted loss for minority classes.

Topics

• DistilBERT
• Fine-tuning
• Text Classification
• Dataset Imbalance
• Model Evaluation
• Weighted F1 Score

Code references

• Sonal12061/distilbert-sentiment-analysis
• Sonal12061/Insurance-classifier

Best for: AI Engineer, Machine Learning Engineer, NLP Engineer, Data Scientist

Related on AIssential

• How I Fine-Tuned DistilBERT to Classify Complaints — And What I Learned Along the Way — Deeply understanding transformer fine-tuning components and their interconnections is crucial for effective model development.
• Every Classification Metric is Just Four Counts — Accuracy alone is a deceptive classification metric; all metrics derive from four confusion matrix counts.
• The Hardest Part of Fine-Tuning Isn’t the Training — The most challenging aspect of NLP fine-tuning is effective label design and meticulous data annotation, not the model training itself.
• What properties of reasoning supervision are associated with improved downstream model quality? — Intrinsic data metrics can reliably predict reasoning dataset utility before model training, with scale-dependent predictors.
• HARP: Efficient Data Selection for Finetuning Large Language Models — HARP efficiently selects finetuning data for LLMs by hierarchically evaluating representative subsets and inferring utilities.
• How to Fine-Tune an SLM for Emotion Recognition — Fine-tuning SLMs for fine-grained emotion recognition on imbalanced datasets requires a multi-technique approach to data balancing and loss weighting.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.