Is Sentiment Analysis in Qualitative Data Analysis Software Accurate?

2024-05-21 · Source: Provalis Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

This analysis evaluates the accuracy and biases of sentiment analysis engines within qualitative data analysis (QDA) software, specifically comparing QDA Miner/WordStat, Atlas.ti, NVivo, and MaxQDA. The study used four benchmarks, including publicly available datasets of online reviews and the SemEval-2013 Twitter benchmark, with 20,000 comments split between very negative and very positive. Initial accuracy scores, ignoring neutral/mixed classifications, ranged from 86.9% to 89.9%. However, the study revealed significant differences in "coverage" (percentage of items classified as positive or negative), with WordStat achieving the highest at 90.9% and NVivo the lowest at 51.4%. When neutral/mixed classifications were reclassified as incorrect, WordStat demonstrated the best overall adjusted accuracy, followed by Atlas.ti, while NVivo and MaxQDA performed poorly. The analysis also identified inherent biases, with MaxQDA consistently showing a strong positive bias and NVivo a clear positive bias, unlike WordStat and Atlas.ti which were closer to balanced. A subsequent update tested Large Language Models (LLMs) like OpenAI, Gemini, Claude, Mistral, and Perplexity, finding they significantly outperformed traditional QDA tools in accuracy, often achieving 99.4-99.6% on review datasets and 82.6-89.2% on the SemEval-2013 benchmark, though at a higher cost and slower processing speed.

Key takeaway

For data scientists and researchers evaluating sentiment analysis tools, recognize that initial high accuracy scores can be deceptive; always scrutinize the tool's classification coverage and inherent biases. While traditional QDA software like WordStat offers transparency and customization, Large Language Models (LLMs) now provide significantly higher accuracy, albeit with increased computational cost and processing time. You should weigh the trade-offs between accuracy, cost, and speed, potentially integrating LLM-based features for critical, high-accuracy tasks.

Key insights

Sentiment analysis accuracy in QDA tools varies significantly, with LLMs offering superior performance at higher costs.

Principles

• High accuracy scores can be misleading without considering classification coverage.
• Sentiment engines often exhibit inherent biases towards positive or negative classifications.
• Customization is crucial for reliable sentiment analysis across diverse domains.

Method

The study created benchmarks from review datasets and Twitter data, classifying extreme sentiments. It assessed accuracy, coverage, and bias, and used Spearman Rho correlation for multi-point ratings.

In practice

• Evaluate sentiment tools beyond headline accuracy by checking coverage and bias.
• Consider LLMs for sentiment analysis when high accuracy is paramount.
• Customize sentiment dictionaries for domain-specific applications.

Topics

• Sentiment Analysis
• Qualitative Data Analysis Software
• Large Language Models
• Text Analytics
• Performance Benchmarking

Best for: Data Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Consistency Analysis of Sentiment Predictions using Syntactic & Semantic Context Assessment Summarization (SSAS) — SSAS improves LLM sentiment prediction consistency by pre-processing noisy data into structured, high-signal prompts.
• Python Sentiment Analysis: From Simple Tools to BERT — Effective sentiment analysis requires matching the right Python tool to text complexity and business needs.
• I spent 4 days training an AI model to understand my country. It barely works — Low-resource NLP projects face significant challenges in data collection, translation, and accurate labeling, often yielding modest baselines.
• Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest — Modern LLMs were systematically evaluated across key social media analytics tasks using a Twitter (X) dataset.
• Use of Machine Learning techniques and Large Language Models for automatic evaluation of Celpe-Bras exam texts — BERT adaptations achieved the best automatic evaluation for Celpe-Bras texts, but at a higher computational cost.
• The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text — Input text characteristics limit LLM prediction accuracy more than prompt or model choice.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Provalis Research.