Proofbusters at SemEval-2026 Task 11: Neuro-Symbolic Syllogistic Reasoning via LLM-Guided Structure Extraction and Deterministic Validation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

The Proofbusters system, developed for SemEval-2026 Task 11, addresses English syllogism validity classification by enabling language models to perform formal reasoning independent of semantic content, thereby mitigating belief bias. Its core approach is symbolic abstraction, converting syllogisms into content-invariant logical forms. Three strategies were explored: Template abstraction (82.20% accuracy), Symbolic OOP abstraction (88.84% with Qwen-7B), and Set-theoretic abstraction. The system employs a two-stage pipeline for formulation (natural language to symbolic) and solving. The set-theoretic variant, utilizing Gemini Flash 2.5 for formulation and Gemini Pro 2.5 for solving, achieved 98.95% accuracy, a 2.13 total content effect, and an overall score of 46.23. This indicates that belief bias is tied to semantic surface form, suggesting robust logical reasoning requires architectural separation between semantic parsing and formal inference.

Key takeaway

For NLP Engineers developing logical reasoning systems, you should prioritize explicit symbolic abstraction to mitigate belief bias and achieve high accuracy in formal tasks like syllogism validity classification. Consider implementing a two-stage neuro-symbolic pipeline, separating semantic parsing from formal inference, rather than relying solely on prompt engineering for robust performance. This approach, exemplified by the 98.95% accuracy of the set-theoretic variant, offers a path to more reliable AI reasoning.

Key insights

Explicit symbolic abstraction enables LLMs to perform formal syllogistic reasoning, overcoming belief bias by reasoning over structure.

Principles

• Belief bias in LLMs stems from semantic surface form.
• Architectural separation improves formal inference.
• Content-invariant logical forms enhance reasoning.

Method

A two-stage pipeline: Formulation converts natural language syllogisms into symbolic representations, followed by a Solver stage that judges validity purely from these symbols.

In practice

• Use set-theoretic abstraction for formal logic tasks.
• Employ Gemini Flash 2.5 for NL-to-symbolic conversion.
• Separate semantic parsing from formal inference.

Topics

• Neuro-Symbolic AI
• Syllogistic Reasoning
• Large Language Models
• Belief Bias
• Formal Logic
• SemEval
• Set Theory

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• System 3 AI: No Humans Needed — System 3 AI integrates formal logic and verifiable code into LLMs for autonomous, complex reasoning beyond human feedback.
• Structured Abductive-Deductive-Inductive Reasoning for LLMs via Algebraic Invariants — A symbolic scaffold improves LLM logical reasoning by separating inference types and enforcing consistency via algebraic invariants.
• Data-driven Machine Learning Cannot Reach Symbolic-level Logical Reasoning -- The Limit of the Scaling Law — Data-driven ML struggles with symbolic logical reasoning due to inherent data and architectural limitations, despite high accuracy.
• FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction — Hybrid neuro-symbolic systems can enhance syllogistic validity prediction by mitigating content bias.
• Grounding vs. Compositionality: On the Non-Complementarity of Reasoning in Neuro-Symbolic Systems — Symbol grounding alone is insufficient for compositional generalization; explicit reasoning objectives are required.
• Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoning — LLM logical reasoning collapses abruptly at critical complexity thresholds, mirroring physical phase transitions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.