Evaluating Robustness of Reasoning Models on Parameterized Logical Problems

2026-02-13 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computational Logic & Reasoning · Depth: Advanced, quick

Summary

A new diagnostic benchmark for 2-SAT, built from parameterized families of structured 2-CNF formulas, has been introduced to evaluate Large Language Model (LLM)-based reasoners. This benchmark addresses the limitation of standard SAT-style tests that often conflate surface difficulty with the structural properties determining satisfiability. The generators isolate specific competencies and failure modes, including contradiction-cycle UNSAT cores, SAT instances with controlled solution multiplicity, planted backbones for propagation modulation, late bridge clauses to test sensitivity to ordering, and symmetry/duplication variants for abstraction testing. The evaluation quantifies decision accuracy, assignment validity, and robustness under semantics-preserving perturbations like clause reordering and variable renaming, revealing sharp performance transitions in LLMs under targeted structural interventions.

Key takeaway

For research scientists developing or evaluating LLM-based reasoners, you should move beyond aggregate SAT accuracy and incorporate structurally-aware benchmarks. Focusing on specific structural interventions, such as varying contradiction-cycle sizes or introducing late bridge clauses, will reveal brittleness regimes and provide a more nuanced understanding of your model's reasoning capabilities and limitations.

Key insights

A new 2-SAT benchmark reveals LLM reasoning brittleness under targeted structural changes, not just surface difficulty.

Principles

• Structural properties, not surface features, determine satisfiability.
• LLM reasoning exhibits sharp performance transitions.
• Semantics-preserving perturbations test robustness.

Method

The method involves generating parameterized 2-CNF formulas with tunable structural axes, isolating specific competencies like contradiction cycles, solution multiplicity, and propagation modulation, then evaluating LLM decision accuracy and assignment validity.

In practice

• Use structured 2-CNF formulas for LLM reasoning tests.
• Vary contradiction-cycle size to test UNSAT core handling.
• Introduce late bridge clauses to probe ordering sensitivity.

Topics

• LLM Reasoning
• 2-SAT Benchmarking
• 2-CNF Formulas
• Implication Graphs
• Structural Robustness

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning — LLMs achieve high accuracy in legal reasoning benchmarks but often lack true logical faithfulness, exhibiting critical failure modes.
• Structured Abductive-Deductive-Inductive Reasoning for LLMs via Algebraic Invariants — A symbolic scaffold improves LLM logical reasoning by separating inference types and enforcing consistency via algebraic invariants.
• QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation — QMFOL provides a verifiable framework for generating logically complex, semantically diverse deductive reasoning benchmarks for LLMs.
• Dissecting Failure Dynamics in Large Language Model Reasoning — LLM reasoning failures often stem from early, high-entropy transition points, not uniform error distribution.
• Structured Abductive-Deductive-Inductive Reasoning for LLMs via Algebraic Invariants — A symbolic reasoning scaffold improves LLM logical consistency by separating inference modes and enforcing algebraic invariants.
• Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs — LLMs exhibit "moral deliberative sycophancy" in non-verifiable reasoning, aligning justifications with user views.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.