From Wordle to Fibble5: Evaluating LLM Reasoning Under Escalating Deception

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

WordleArenas, a new benchmark platform, evaluates large language model (LLM) reasoning robustness under escalating deception, moving beyond standard benchmarks that assume truthful feedback. Generalizing from Wordle and Fibble, the platform introduces Fibblek, allowing for k (0-5) lies per feedback row. The evaluation of 41 models from 10 providers across 3,749 games revealed significant findings: even one lie per row caused catastrophic performance drops (average win rate fell from 41.1% to 18.7%). A sharp "deception cliff" emerged at 2-3 lies, where most models collapsed to a ≤3% win rate. Standard benchmark rankings poorly predicted model robustness to deception. Surprisingly, some models showed partial recovery in Fibble5 (all feedback lies), achieving an average 9.5% win rate, outperforming Fibble3 (0.3%) and Fibble4 (0.4%), because knowing all feedback is false restores deterministic information.

Key takeaway

For ML engineers deploying LLMs in real-world scenarios, recognize that standard benchmark scores significantly overstate model reasoning capabilities under even slight deception. You should integrate deception-aware evaluations, like WordleArenas, into your testing protocols to accurately assess robustness. Prioritize models that demonstrate resilience to misinformation, especially when feedback reliability is uncertain, to prevent catastrophic performance drops and ensure reliable application behavior.

Key insights

LLM reasoning robustness is severely degraded by even minimal deception, a factor poorly captured by standard benchmarks.

Principles

• Truthful-feedback evaluations systematically overestimate LLM capabilities.
• Deception-aware benchmarks are essential for assessing real-world robustness.
• Deterministic information, even if fully deceptive, can aid LLM performance.

Method

WordleArenas evaluates LLM reasoning robustness by generalizing Wordle to Fibblek, where k (0-5) represents lies per feedback row, across 3,749 games.

In practice

• Integrate deception-aware benchmarks like WordleArenas into LLM testing.
• Analyze LLM performance under "all-lie" scenarios for deterministic deceptive feedback.

Topics

• LLM Evaluation
• Reasoning Robustness
• Deception Benchmarks
• Misinformation
• WordleArenas
• Fibblek

Best for: Research Scientist, AI Engineer, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks — LLMs exhibit significant arithmetic reasoning fragility even with small, schema-preserving numeric changes, particularly on complex datasets like GSM8K.
• Claude Opus 4.8: Lying Machine No More? — Claude Opus 4.8 prioritizes honesty and reliability over raw benchmark scores, addressing prior deception and laziness.
• OWASP LLM09:2025 Misinformation — LLM misinformation, driven by hallucination and overreliance, poses significant security, reputational, and legal risks.
• Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning — LLMs achieve high accuracy in legal reasoning benchmarks but often lack true logical faithfulness, exhibiting critical failure modes.
• Evaluating Counterfactual Strategic Reasoning in Large Language Models — LLMs struggle with genuine strategic reasoning in counterfactual game environments, often relying on memorized patterns.
• UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs — LLMs struggle significantly with generating samples calibrated to true underlying distributions, hindering their use in simulations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.