GUIR at SemEval-2026 Task 7: Probing Cultural Knowledge in LLMs via Multi-Agent Debate

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

The GUIR system, developed for SemEval-2026 Task 7, investigates the cultural knowledge encoded in general-purpose Large Language Models without culture-specific supervision. Addressing two tracks on the BLEnD benchmark, the system achieved 55.5% accuracy on the short-answer question (SAQ) track across 61 language locales using zero-shot prompting with gpt-4.1. For the multiple-choice question (MCQ) track, a three-stage pipeline was implemented, involving zero-shot chain-of-thought inference with gpt-5-mini, cross-locale majority voting, and a multi-agent debate protocol. This pipeline secured a 97.47% overall accuracy across 30 locales, ranking first among all submissions. Furthermore, a human evaluation on the Persian locale revealed that BLEnD's lemma-matching scorer underestimated model performance by 18 percentage points, highlighting a need for improved evaluation methods for morphologically rich languages.

Key takeaway

For NLP engineers evaluating LLM performance on culturally diverse datasets, you should consider implementing multi-agent debate protocols to enhance accuracy, especially for multiple-choice tasks. Your current lemma-matching evaluation methods might be underestimating true model capabilities, particularly for morphologically rich languages like Persian. Incorporate human evaluation to validate scores and ensure robust assessment of cultural knowledge.

Key insights

Multi-agent debate significantly enhances LLM cultural knowledge assessment, outperforming standard evaluation metrics.

Principles

• LLMs encode cultural knowledge without explicit fine-tuning.
• Multi-agent debate improves LLM accuracy on complex tasks.
• Lemma-matching can misrepresent performance in rich languages.

Method

A three-stage pipeline for MCQ involves zero-shot chain-of-thought inference, cross-locale majority voting for consistency, and a multi-agent debate protocol to resolve residual errors.

In practice

• Implement multi-agent debate for complex LLM reasoning tasks.
• Use cross-locale voting to stabilize LLM predictions.
• Conduct human evaluation for morphologically rich language tasks.

Topics

• LLM Cultural Knowledge
• Multi-Agent Debate
• SemEval-2026 Task 7
• BLEnD Benchmark
• Cross-Locale Evaluation
• Morphologically Rich Languages

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual and multicultural reasoning in grounded, real-world tasks.
• Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages — ILR-informed expert judgment reveals interpretable, domain-dependent cross-lingual variation in LLM outputs.
• GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations — LLM agent memory systems perform poorly in multi-party conversations, often outperformed by simple baselines.
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions — Multi-Legal-Bench reveals LLM legal reasoning varies significantly across jurisdictions and tasks, with transfer quality tied to label-set alignment.
• Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency — LLMs can self-align cultural knowledge across languages by identifying consistent responses and transferring them to weaker language contexts.
• Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants — LLMs exhibit significant performance gaps in culturally-grounded, dialectal Arabic, especially with open-ended questions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.