dutirshlee at SemEval-2026 Task 11: Symbolic Augmentation for Content-Bias-Resistant Syllogistic Reasoning

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

dutirshlee's system for SemEval-2026 Task 11 Subtask 1 addresses English syllogistic validity using a fine-tuned Qwen2.5-7B-Instruct model. The core innovation is Symbolic Data Augmentation (SDA), which replaces real-world entities with abstract placeholders to explicitly separate logical form from content, thereby enhancing resistance to content bias. This approach achieved 96.34% accuracy and a Total Content Effect (TCE) of 2.15, resulting in a primary score of 44.86. Ablation studies confirmed that direct LoRA training with SDA is the most robust configuration, with other methods like prompting and DPO yielding negative results. Furthermore, a specialist-generalist complementarity setting demonstrated that an API model (99.48% ACC, 1.06 TCE) corrected by the SDA specialist on disagreements could achieve 100% accuracy and 0 TCE.

Key takeaway

For NLP Engineers developing robust reasoning systems, this work suggests prioritizing Symbolic Data Augmentation (SDA) with direct fine-tuning. Your models can achieve high accuracy and zero content bias by explicitly decoupling logical form from content. Consider a specialist-generalist ensemble, where an SDA-trained model corrects a powerful API model, to maximize performance and eliminate content effects in syllogistic reasoning tasks.

Key insights

Symbolic Data Augmentation effectively decouples logical form from content, improving syllogistic reasoning and reducing content bias in LLMs.

Principles

• Decouple logic from content for robust reasoning.
• Specialist-generalist model ensembles enhance performance.
• Direct fine-tuning can outperform complex prompting.

Method

Fine-tune Qwen2.5-7B-Instruct with LoRA using Symbolic Data Augmentation (SDA). SDA replaces real-world entities with abstract placeholders to remove content bias.

In practice

• Apply SDA to train LLMs for logical tasks.
• Combine a strong API model with a specialist.
• Prioritize direct fine-tuning over prompting.

Topics

• Syllogistic Reasoning
• Large Language Models
• Data Augmentation
• Content Bias
• Qwen2.5-7B-Instruct
• SemEval-2026

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction — Hybrid neuro-symbolic systems can enhance syllogistic validity prediction by mitigating content bias.
• When Does Synthetic Data Augmentation Improve Score-Based Imbalanced Classification? — Synthetic data augmentation improves imbalanced classification only under model misspecification, not well-specification.
• Reasoning Quality Emerges Early: Data Curation for Reasoning Models — Reasoning quality in LLMs can be effectively improved by curating SFT data based on early token loss patterns, significantly boosting efficiency.
• Negation-Aware Data Augmentation for Portuguese Natural Language Inference — Targeted negation-aware data augmentation improves BERT-based NLI model performance on Portuguese datasets by reducing negation bias.
• Symbol-Equivariant Recurrent Reasoning Models — Explicitly encoding symbol symmetry in neural networks enhances reasoning model robustness and scalability.
• Data-driven Machine Learning Cannot Reach Symbolic-level Logical Reasoning -- The Limit of the Scaling Law — Data-driven ML struggles with symbolic logical reasoning due to inherent data and architectural limitations, despite high accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.