GIL-Zaragoza at SemEval 2026 Task 11: Comparing Classification, Autoformalization, and Ontologies for Formal Reasoning Capabilities

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

GIL-Zaragoza participated in SemEval-2026 Task 11, which assessed models' ability to determine the logical validity of syllogisms without real-world content. The team developed and compared three distinct approaches for Subtask 1. These included an encoder-based classification baseline utilizing classical ML and fine-tuned BERT with debiasing strategies, an autoformalization pipeline integrating DPO-aligned models with first-order logic translation and formal inference via Prover9, and a hybrid neuro-symbolic method employing GPT to generate OWL 2 ontologies evaluated by the HermiT reasoner. The encoder-based classifier achieved the best performance, securing a 72.25% accuracy and a combined score of 20.37, placing 40th among 45 participating teams. Analysis revealed classification methods had lower content bias, while autoformalization faced translation issues, and ontology-based reasoning struggled with prompt design and verbose formats.

Key takeaway

For NLP engineers developing formal reasoning systems, you should prioritize simpler, direct classification models like fine-tuned BERT for tasks like syllogism validity. While neuro-symbolic and autoformalization approaches show promise, their current limitations in translation consistency and prompt design make them less reliable. Focus on debiasing strategies within classification to improve performance, as demonstrated by the 72.25% accuracy achieved. Your efforts are better spent refining these robust methods than battling complex formalization pipeline inconsistencies.

Key insights

Direct classification methods currently outperform autoformalization and ontology-based reasoning for syllogism validity.

Principles

• Content bias impacts logical reasoning.
• Formalization pipelines introduce inconsistencies.
• Prompt design limits neuro-symbolic systems.

Method

The study compared encoder-based classification, DPO-aligned autoformalization with Prover9, and GPT-generated OWL 2 ontologies with HermiT for syllogism validity.

In practice

• Prioritize encoder-based classifiers for syllogism tasks.
• Address translation issues in autoformalization.
• Refine prompt engineering for ontology generation.

Topics

• SemEval-2026 Task 11
• Syllogism Validity
• Formal Reasoning
• Encoder-based Classification
• Autoformalization
• Neuro-symbolic AI

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction — Hybrid neuro-symbolic systems can enhance syllogistic validity prediction by mitigating content bias.
• Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation — LLMs achieve only 8.6% semantic correctness for TLA+ specifications, primarily with progressive prompting.
• Natural Language Autoencoders: Turning Claude’s thoughts into text — Natural Language Autoencoders translate AI model internal activations into human-readable text, revealing hidden thoughts and motivations.
• What properties of reasoning supervision are associated with improved downstream model quality? — Intrinsic data metrics can reliably predict reasoning dataset utility before model training, with scale-dependent predictors.
• mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection — Data augmentation and self-training can effectively finetune LLMs for specialized text classification tasks with limited data.
• The Faithfulness Gap: Certifying Semantic Equivalence Between Natural-Language and Formal Mathematical Statements — Certifying semantic equivalence between natural language and formal math statements is crucial for autoformalization faithfulness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.