Controlled Reformulation Testing for Logical Consistency in Large Language Models

2026-07-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

CRTBench introduces a new benchmark of 350 question families, totaling 1,750 questions, designed to evaluate logical invariance in large language models (LLMs). This benchmark assesses LLMs' ability to maintain consistent answers across controlled reformulations, including contrapositive rewriting, double negation, negation flipping, and passive voice. Experiments reveal a significant accuracy-consistency gap; for instance, GPT-5.4-mini achieves 98.9% base accuracy but only 60.3% family-level consistency. In contrast, the reasoning-optimized o4-mini demonstrates 96.9% consistency. Failures predominantly occur with logically nontrivial transformations like contrapositive rewriting (72.4% for GPT-5.4-mini) and double negation (84.6%), while surface-level rephrasing remains robust at 94-100%. Increasing reasoning effort improved GPT-5.4-mini's consistency to 85.4%, but GPT-5.4 remained unchanged due to offsetting gains and failures. These findings underscore that base accuracy alone is insufficient for evaluating LLM logical reasoning.

Key takeaway

For AI Scientists evaluating LLMs for critical applications, relying solely on base accuracy is insufficient. You must incorporate logical consistency testing, especially for nontrivial reformulations like contrapositives and double negations, to truly assess reasoning capabilities. Consider models optimized for reasoning, such as o4-mini, which demonstrate higher consistency, to mitigate risks of contradictory outputs in production.

Key insights

LLMs struggle with logical consistency across equivalent question reformulations despite high base accuracy.

Principles

• Logical invariance is distinct from base accuracy in LLM evaluation.
• Nontrivial logical transformations reveal consistency gaps more than surface rephrasing.
• Reasoning effort can improve consistency but may have trade-offs.

Method

CRTBench evaluates LLM logical invariance using 350 question families (1,750 questions) across controlled reformulations like contrapositive rewriting, double negation, negation flipping, and passive voice.

In practice

• Test LLMs for consistency on contrapositive rewriting and double negation.
• Evaluate models like GPT-5.4-mini and o4-mini for logical invariance.

Topics

• Large Language Models
• Logical Consistency
• LLM Evaluation
• Benchmarking
• Natural Language Processing
• Reasoning

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Reliable Chain-of-Thought via Prefix Consistency — Truncating and regenerating LLM Chain-of-Thought traces reveals a reliability signal for improved answer aggregation.
• Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoning — LLMs can produce unfaithful logical proofs despite high compilation rates, highlighting a gap between validity and faithfulness.
• Semantic Invariance in Agentic AI — LLM reasoning stability under semantic input variations is crucial for reliable autonomous agents.
• From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text — Evaluating LLMs for legal text requires both quantitative benchmarks and qualitative error analysis to reveal true capabilities.
• BREAKING: LLM “reasoning” continues to be deeply flawed — LLMs, even those marketed for reasoning, continue to exhibit fundamental and widespread reasoning failures.
• Trust Me. I’m Artificial Intelligence. — AI models are becoming more convincing in their reasoning, but this does not equate to increased truthfulness or correctness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.