Reasoning or Memorization: Can LLMs Understand and Generate Chinese Xiehouyu Riddles?

2026-07-26 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A study investigated large language models' (LLMs) reasoning versus memorization capabilities using Chinese xiehouyu riddles, employing novel riddles created by linguists to prevent data contamination. The research evaluated LLMs through multiple-choice questions (MCQ), free-form explanation generation, and new xiehouyu creation. A key metric, $Δ_{acc}$ (delta of accuracy between existing low-frequency and novel xiehouyu), indicated memorization. Frontier Chinese models showed an average $Δ_{acc}$ of 23.6%, implying extensive training on Chinese data, while English-centric models had a mean $Δ_{acc}$ of 5.1%. Notably, Gemini 3.1 Pro achieved 92.6% accuracy on novel xiehouyu, surpassing human accuracy by 24%. However, LLM-generated xiehouyu received significantly lower ratings than human creations. These findings suggest a need to re-examine LLM reasoning claims, considering potential data contamination, and highlight that LLM creativity in complex language tasks like Chinese xiehouyu may still trail human expertise.

Key takeaway

For NLP Engineers evaluating LLM capabilities in nuanced language tasks, you should critically assess reasoning claims by using novel, uncontaminated data. Your benchmarks should include metrics like $Δ_{acc}$ to distinguish true reasoning from memorization, particularly for models trained on extensive non-English datasets. While models like Gemini 3.1 Pro show strong performance on novel understanding, recognize that current LLMs may still fall short of human creativity in generating complex linguistic forms like xiehouyu.

Key insights

LLMs demonstrate strong memorization and some reasoning for novel riddles, but lack human-level creativity in generating complex language.

Principles

• $Δ_{acc}$ can index LLM memorization.
• Data contamination impacts reasoning claims.
• Creativity lags human experts in complex language.

Method

Evaluated LLMs on Chinese xiehouyu using MCQs, free-form explanations, and creation tasks, with novel riddles and $Δ_{acc}$ to differentiate memorization from reasoning.

In practice

• Use $Δ_{acc}$ for LLM memorization assessment.
• Prioritize novel data for reasoning tests.
• Benchmark LLM creativity against human experts.

Topics

• LLM Reasoning
• Memorization
• Chinese Xiehouyu
• Data Contamination
• Language Generation
• Gemini 3.1 Pro

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Can LLMs Solve My Grandma’s Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles — LLMs perform significantly below human levels on culturally grounded, figurative Bangla riddles, highlighting a gap in low-resource reasoning.
• Why isn’t LLM reasoning done in vector space instead of natural language?[D] — LLMs internally use vectors, but natural language reasoning offers crucial interpretability and debuggability.
• Code, Chaos, and the Invisible Logic of LLMs — LLMs predict words probabilistically, not with human intention, leading to potential misunderstandings in code generation.
• Thinking to recall: How reasoning unlocks parametric knowledge in LLMs — Reasoning in LLMs unlocks parametric knowledge through latent computation and factual priming, even for simple facts.
• The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans — LLMs struggle with flexible reasoning, often conflating pattern matching with genuine understanding, unlike humans who adapt reasoning strategies.
• Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity — LLM pragmatic reasoning is an interaction between internal representations and task-induced behavior, not stable competence.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.