Can LLMs Solve My Grandma’s Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

BanglaRiddleEval is a new benchmark designed to assess Large Language Models' reasoning abilities on 1,244 traditional Bangla riddles, comprising 4,976 riddle-task artifacts across four distinct tasks. Researchers utilized an LLM-based pipeline to generate Chain-of-Thought explanations, semantically coherent distractors, and fine-grained ambiguity annotations. A diverse suite of open-source and closed-source models were evaluated using various prompting strategies. Results indicate that models achieved moderate semantic overlap on generative QA but low correctness. MCQ accuracy peaked at approximately 56%, significantly below the 83.3% human baseline. Ambiguity resolution performance ranged from roughly 26% to 68%, with only the strongest models producing high-quality explanations. These findings demonstrate that current LLMs capture some necessary cues for Bangla riddle reasoning but remain considerably below human-level performance, positioning BanglaRiddleEval as a challenging benchmark for low-resource figurative reasoning.

Key takeaway

For NLP Engineers developing or deploying LLMs in low-resource, culturally-specific contexts, this research highlights significant performance gaps. Your models will likely struggle with figurative reasoning tasks like traditional riddles, achieving only about 56% accuracy compared to human baselines. You should prioritize research and development into improving LLM understanding of cultural nuances and ambiguity resolution to bridge this substantial performance disparity.

Key insights

LLMs perform significantly below human levels on culturally grounded, figurative Bangla riddles, highlighting a gap in low-resource reasoning.

Principles

• LLMs capture some figurative reasoning cues.
• Cultural context is key for human-level reasoning.

Method

An LLM-based pipeline generated Chain-of-Thought explanations, distractors, and ambiguity annotations for 1,244 riddles. Diverse models were evaluated using different prompting strategies.

In practice

• Utilize BanglaRiddleEval for low-resource reasoning.
• Develop models for cultural context and ambiguity.

Topics

• Large Language Models
• NLP Benchmarks
• BanglaRiddleEval
• Low-Resource Languages
• Figurative Reasoning
• Cultural Context

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting interpretation accuracy.
• The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans — LLMs struggle with flexible reasoning, often conflating pattern matching with genuine understanding, unlike humans who adapt reasoning strategies.
• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual and multicultural reasoning in grounded, real-world tasks.
• Textual Inference in Portuguese: Comparing Language Models — LLMs struggle with logic-sensitive semantic reasoning in Portuguese, highlighting the need for specialized benchmarks.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs reason effectively across 43 languages, with machine translation not degrading accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.