Takoyaki at SemEval-2026 Task 3: Ensembling LLM Predictions using Demonstration Retrieval for Dimensional Aspect-based Sentiment Analysis

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, medium

Summary

The "Takoyaki" system, presented at SemEval-2026 Task 3 (DimABSA), addresses Dimensional Aspect-based Sentiment Analysis, specifically Subtask 2 (DimASTE) for extracting aspect term, opinion term, and valence-arousal triplets, and Subtask 3 (DimASQP) for adding aspect category classification to form quadruplets. The proposed multi-step pipeline integrates retrieval-based in-context learning using BM25 for demonstration selection, an agreement-based ensemble of LLM predictions from multiple retrieval variants, and, for certain datasets like Japanese, error-pattern correction. Retrieval-based ICL and the ensemble consistently improved performance across languages and domains, with error correction further enhancing results for the Japanese dataset. Human evaluation revealed that LLM-based labeling achieved higher agreement with gold labels than human annotators, indicating a notable discrepancy between automated metrics and practical output quality.

Key takeaway

For NLP Engineers developing advanced sentiment analysis systems, consider integrating a multi-step LLM pipeline. You should implement retrieval-based in-context learning with BM25 for demonstration selection and an agreement-based ensemble to combine LLM predictions, as these consistently improve results. Furthermore, for specific language datasets like Japanese, applying error-pattern correction can yield significant gains. Your evaluation should also include human assessment, as automated metrics may not fully reflect practical output quality.

Key insights

Ensembling LLM predictions with retrieval-based in-context learning and error correction improves dimensional aspect-based sentiment analysis, often outperforming human annotators.

Principles

• Retrieval-based ICL consistently improves LLM performance.
• Ensembling LLM predictions enhances robustness.
• LLM-based labeling can surpass human annotator agreement.

Method

A multi-step pipeline: (1) BM25-based retrieval for LLM in-context learning, (2) agreement-based ensemble of LLM predictions, (3) error-pattern correction using rule sets for uncertain outputs.

In practice

• Apply BM25 for relevant demonstration selection.
• Combine multiple LLM retrieval variants via ensembling.
• Implement rule-based error correction for specific languages.

Topics

• Dimensional Aspect-based Sentiment Analysis
• Large Language Models
• In-Context Learning
• Ensemble Methods
• BM25 Retrieval
• SemEval-2026 Task 3
• Human Evaluation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Consistency Analysis of Sentiment Predictions using Syntactic & Semantic Context Assessment Summarization (SSAS) — SSAS improves LLM sentiment prediction consistency by pre-processing noisy data into structured, high-signal prompts.
• AspectRAG: An Architecture of Retrieval and Generation for Aspect-Based Sentiment Analysis — AspectRAG leverages aspect-guided retrieval and generation for unsupervised, high-performance sentiment analysis in Portuguese.
• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• The State of Information Retrieval in 2026 — Information retrieval has evolved into a reasoning problem, integrating LLMs and agentic search for complex, context-aware query processing.
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions — LLMs achieve high semantic alignment through paraphrasing, not lexical reproduction, challenging traditional evaluation metrics.
• Zero-Shot to Full-Resource: Cross-lingual Transfer Strategies for Aspect-Based Sentiment Analysis — Cross-lingual transfer strategies for ABSA vary in effectiveness based on model architecture and resource availability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.