pamaldi at SemEval-2026 Task 11: Neuro-Symbolic Syllogistic Reasoning via LLM-Guided Structure Extraction and Deterministic Validation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

pamaldi's system, participating in SemEval-2026 Task 11, Subtask 1, addresses the formal validity of English syllogisms while minimizing content plausibility. This neuro-symbolic pipeline uses an LLM to extract formal structure, including proposition types (A, E, I, O) and terms. The syllogistic figure is deterministically computed, and a symbolic validator checks against 24 classically valid Aristotelian forms. The system achieved 96.34% accuracy, a Total Content Effect (TCE) of 1.02, and a combined score of 56.57. This significantly outperforms pure-LLM baselines, more than doubling their combined score from 26.52 and reducing TCE by nearly an order of magnitude. Swapping the extractor to Claude Sonnet 4.5 maintained performance, confirming the symbolic stage's role in content-invariance. A paraphrase probe identified extractor sensitivity to premise ordering as a key limitation.

Key takeaway

For AI Scientists and NLP Engineers designing robust reasoning systems, this work demonstrates that neuro-symbolic approaches significantly improve accuracy and reduce content bias in logical reasoning tasks compared to pure-LLM methods. You should consider integrating deterministic symbolic validation stages to enhance reliability and interpretability in your LLM-based reasoning pipelines, especially for tasks requiring formal correctness over plausibility.

Key insights

A neuro-symbolic pipeline effectively separates LLM-based structure extraction from deterministic symbolic validation for syllogistic reasoning.

Principles

• Strict separation of neural and symbolic components enhances robustness.
• Deterministic validation ensures content-invariance in reasoning tasks.
• LLMs excel at formal structure extraction from natural language.

Method

The system uses an LLM to extract proposition types (A, E, I, O) and terms, deterministically computes the syllogistic figure, then a symbolic validator checks against 24 valid Aristotelian forms.

In practice

• Apply LLMs for initial natural language parsing into formal structures.
• Implement symbolic validators for logical consistency checks.
• Probe system components for specific sensitivities like premise ordering.

Topics

• Neuro-Symbolic AI
• Syllogistic Reasoning
• LLM Structure Extraction
• SemEval-2026 Task 11
• Formal Logic Validation
• Content Invariance

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction — Hybrid neuro-symbolic systems can enhance syllogistic validity prediction by mitigating content bias.
• System 3 AI: No Humans Needed — System 3 AI integrates formal logic and verifiable code into LLMs for autonomous, complex reasoning beyond human feedback.
• Structured Abductive-Deductive-Inductive Reasoning for LLMs via Algebraic Invariants — A symbolic scaffold improves LLM logical reasoning by separating inference types and enforcing consistency via algebraic invariants.
• Neuro-Symbolic Software Verification: Hyper-charging Local Language Models with Symbolic Reasoning at Scale — Neuro-symbolic verification with local LLMs effectively synthesizes loop invariants, enhancing privacy and efficiency over cloud APIs.
• Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoning — LLM logical reasoning collapses abruptly at critical complexity thresholds, mirroring physical phase transitions.
• Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law — LLM legal reasoning performance is inflated by data contamination; neuro-symbolic systems offer superior robustness and generalization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.