Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go?

2026-04-16 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

A study investigated the efficacy of zero-shot learning (ZSL) for sentiment analysis in software engineering, aiming to overcome the challenge of scarce annotated datasets. Researchers evaluated embedding-based, natural language inference (NLI)-based, task-aware representation of sentences (TARS)-based, and generative-based ZSL techniques across seven publicly available software engineering datasets, including API reviews, GitHub comments, and Jira issues. The study assessed the impact of various label configurations (original, expert-curated, LLM-generated) and compared ZSL performance against state-of-the-art fine-tuned transformer-based models. Findings indicate that ZSL techniques, particularly embedding-based models like E_M9 combined with expert-curated labels, can achieve macro-F1 scores comparable to or exceeding fine-tuned models. Error analysis revealed that subjectivity in annotation and "polar facts" were primary causes of misclassifications, especially for neutral sentiments.

Key takeaway

For AI Engineers developing sentiment analysis tools in software engineering, consider implementing ZSL with pre-trained embedding-based models like E_M9 and expert-curated labels. This approach can yield performance competitive with fine-tuned models, significantly reducing the need for costly, context-specific annotated datasets. Be aware that neutral sentiment classification remains a challenge, often due to annotation subjectivity and "polar facts," requiring careful post-processing or targeted refinement.

Key insights

Zero-shot learning offers a viable solution for sentiment analysis in software engineering, mitigating annotated data scarcity.

Principles

• ZSL can match fine-tuned models without additional training.
• Expert-curated labels enhance ZSL performance.
• Neutral sentiments are challenging for ZSL models.

Method

The study empirically evaluated four ZSL techniques (embedding, NLI, TARS, generative) with varied label configurations (original, expert-curated, LLM-generated) on seven software engineering datasets, comparing macro-F1 scores against fine-tuned transformers.

In practice

• Prioritize expert-curated labels for ZSL sentiment tasks.
• Explore freely available ZSL models before paid options.
• Focus error analysis on neutral sentiment classifications.

Topics

• Sentiment Analysis
• Software Engineering
• Zero-shot Learning
• Natural Language Processing
• Transformer Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go? — Zero-Shot Learning can achieve competitive sentiment analysis in software engineering, reducing reliance on annotated datasets.
• Using a Local LLM as a Zero-Shot Classifier — Locally hosted LLMs can effectively perform zero-shot classification on short, semantically complex texts where traditional methods fail.
• Zero-Shot NER with GliNER and spaCy — GliNER enables zero-shot named entity recognition for both generic and domain-specific labels without training data.
• On GATE, Text and Social Media Analysis, and Detecting Misinformation Online — A multi-layer analytical pipeline effectively extracts and categorizes digital identity aspects of agrifood products from online discourse.
• Rating–Text Mismatch in Brazilian Portuguese Reviews: How Reliable Are Zero-Shot LLMs? — LLMs effectively detect rating-text incoherence in Brazilian Portuguese reviews with high F1 scores and strong inter-model agreement.
• Distilling Examples into Task Instructions: Enhanced In-Context Learning for Real-World B2B Conversations — Distilling ICL examples into structured instructions significantly improves performance and efficiency for complex B2B conversation classification.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.