From TF-IDF to Transformers: A Comparative and Ensemble Approach to Sentiment Classification

2026-05-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Advanced, long

Summary

This study presents a comparative and ensemble approach to sentiment classification using the IMDb dataset, comprising 50,000 movie reviews. Researchers evaluated diverse machine learning models, including Naïve Bayes, Logistic Regression, Support Vector Machines (SVM), LightGBM, LSTM, and transformer-based models like RoBERTa and DistilBERT. Data preprocessing involved text normalisation, tokenisation, and TF-IDF vectorization for traditional models, while transformers utilized subword tokenisation and contextual embeddings. RoBERTa achieved the highest individual accuracy at 93.02%. A soft voting ensemble, combining all models with equal weights, further enhanced classification performance, reducing misclassification rates for sarcastic reviews by approximately 23% and showing a 1-3% improvement across metrics compared to RoBERTa alone. The study also employed SHAP for model explainability, attributing predictions to specific tokens.

Key takeaway

For Machine Learning Engineers building sentiment analysis systems, you should prioritize transformer-based models like RoBERTa for maximum accuracy, especially with complex language. To significantly improve robustness and handle nuanced cases like sarcasm, integrate a soft voting ensemble of diverse models. Consider DistilBERT if computational efficiency is a key constraint, as it offers strong performance. Your approach should also include SHAP for crucial model transparency.

Key insights

A soft voting ensemble of diverse machine learning and transformer models significantly enhances sentiment classification accuracy, particularly for nuanced text.

Principles

• Transformer models capture subtle semantic nuances.
• Soft voting ensembles improve robustness and reduce variance.
• TF-IDF with SVM offers an efficient, interpretable baseline.

Method

Text was preprocessed with normalisation, tokenisation, and TF-IDF or subword tokenisation. Seven diverse models (traditional ML, LSTM, transformers) were trained, and their probability predictions were combined via a soft voting ensemble. SHAP explained predictions.

In practice

• Implement soft voting ensembles for robust sentiment analysis.
• Fine-tune RoBERTa for superior contextual understanding.
• Use DistilBERT for strong performance with efficiency.

Topics

• Sentiment Analysis
• Transformer Models
• Ensemble Learning
• RoBERTa
• DistilBERT
• SHAP Explainability

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Can Machines Tell Whether Gamers Love or Hate a Game? — Fine-tuning Transformer models like RoBERTa significantly improves sentiment analysis accuracy on imbalanced, complex human language datasets.
• TeamHerald@CHIPSAL 2026: Hate Speech Detection and Sentiment Analysis of Nepali Memes using Transformer-based Architectures and Ensemble Learning — Ensemble strategies for meme analysis perform differently for binary vs. multi-class tasks, requiring task-specific aggregation.
• Teaching Machines to “Feel” Tweets — Domain-native pre-trained models are crucial for accurate sentiment classification in ambiguous social media language.
• Ideology Prediction of German Political Texts — Transformer models can effectively map German political texts to a continuous ideological spectrum.
• Fine-Tuning BERT for Sentiment Analysis: A Beginner-Friendly Guide — Fine-tuning pre-trained transformer models like BERT enables high-accuracy NLP tasks with minimal training data.
• Automatic Reflection Level Classification in Hungarian Student Essays — Automated reflection classification in Hungarian essays is feasible using both classical ML and fine-tuned transformers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.