Ellat at SemEval-2026 Task 11: Comparing Encoder and Decoder Models for Syllogistic Reasoning

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

Team Ellat investigated language models' ability to assess logical validity independently of semantic plausibility for SemEval-2026 Task 11 (Subtask 1: English). The study evaluated three architectures: MiniLM-L6-mnli-binary, DeBERTa-v3-small, and Llama 3.1-8B-Instruct. Researchers applied task-specific fine-tuning for encoder models and Abstract Logic Augmentation with QLoRA for LLaMA. DeBERTa-v3-small achieved the strongest overall performance, while MiniLM-L6-mnli-binary showed clear reductions in content bias after fine-tuning. Llama 3.1-8B-Instruct exhibited strong plausibility bias in a zero-shot setting. Although fine-tuning and abstraction-based augmentation reduced plausibility bias, fully separating logical validity from semantic content remains challenging across all tested architectures, with augmented fine-tuning yielding only modest improvements.

Key takeaway

For NLP Engineers evaluating language models for formal logical reasoning, recognize that separating logical validity from semantic plausibility remains a significant challenge. While fine-tuning encoder models like MiniLM-L6-mnli-binary can reduce content bias, and Abstract Logic Augmentation with QLoRA offers modest improvements for decoder models like Llama 3.1-8B-Instruct, zero-shot performance often exhibits strong plausibility bias. You should prioritize architectures and fine-tuning strategies that explicitly target structural reasoning over semantic shortcuts to mitigate this inherent limitation.

Key insights

Language models struggle to separate logical validity from semantic plausibility, even with fine-tuning and augmentation.

Principles

• LMs often rely on world knowledge and semantic shortcuts.
• Fine-tuning can reduce content bias in encoder models.
• Zero-shot LMs exhibit strong plausibility bias.

Method

Evaluated MiniLM-L6-mnli-binary, DeBERTa-v3-small, and Llama 3.1-8B-Instruct. Applied task-specific fine-tuning for encoders and Abstract Logic Augmentation with QLoRA for LLaMA to assess syllogistic reasoning.

In practice

• Fine-tune encoder models to reduce content bias.
• Augment decoder models with QLoRA for logic tasks.
• Be aware of strong plausibility bias in zero-shot LMs.

Topics

• Syllogistic Reasoning
• Language Model Evaluation
• Encoder Models
• Decoder Models
• Fine-tuning
• Plausibility Bias

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Decoder only Transformer : Building a GPT-2 model prototype to make it understand Natural Language… — Training a GPT-2 prototype on classic novels demonstrates decoder-only Transformer architecture and performance benefits of pre-trained weights.
• LoRA & QLoRA Mastery: The Beginner-to-Advanced Guide to Efficient LLM Fine-Tuning — LoRA and QLoRA enable efficient LLM fine-tuning by adapting a small fraction of parameters, drastically reducing memory and compute.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs show affirmation bias, with negation understanding varying significantly across diverse languages and models.
• Modeling Pathology-Like Behavioral Patterns in Language Models Through Behavioral Fine-Tuning — Consistent behavioral fine-tuning can induce stable, pathology-like biases in LLM generative distributions and action selection.
• DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition — Simulating human pilot annotation with LLMs to refine instructions significantly boosts zero-shot NER performance.
• More Thinking, More Bias: Length-Driven Position Bias in Reasoning Models — Longer reasoning trajectories in LLMs increase position bias, a critical factor for evaluation reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.