AI4PC-Howard University at SemEval-2026 Task 12: Evidence-Guided Abductive Scoring with Option-Conditioned Retrieval and Constrained LLM Evaluation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

AI4PC-Howard University developed an evidence-guided abductive scoring pipeline for SemEval-2026 Task 12, designed to select plausible explanations for events from noisy, multi-document contexts. This modular system separates evidence selection from explanation scoring. It operates by chunking documents, retrieving option-conditioned evidence using dense embeddings, and then employing a cross-encoder reranker to create compact evidence packs for each explanation option. A constrained large language model (LLM) scorer evaluates each option solely based on its evidence pack, generating structured signals for evidence support, explanatory directness, and contradiction. The pipeline then applies deterministic decision rules to make single or multi-label predictions, including robustly identifying "none of the above" options through lexical-cue detection. This design minimizes irrelevant document distraction, enhances comparability across options, and allows for controlled calibration of multi-answer outputs, supporting abductive reasoning without relying on knowledge graphs or extensive end-to-end LLM prompting.

Key takeaway

For NLP engineers developing abductive reasoning systems, consider adopting a modular, evidence-guided LLM approach. Your systems can achieve greater accuracy and robustness by separating evidence retrieval from explanation scoring. Focus on creating compact, option-conditioned evidence packs for your LLMs and use structured signals to evaluate explanations. This method allows you to handle "none of the above" options effectively and reduces reliance on full document context, streamlining your reasoning pipelines.

Key insights

Modular evidence-guided LLM scoring improves abductive reasoning by separating evidence selection from explanation evaluation.

Principles

• Separate evidence selection from explanation scoring.
• Use compact, option-conditioned evidence for LLM evaluation.
• Employ structured signals for robust explanation assessment.

Method

The pipeline chunks documents, retrieves option-conditioned evidence via dense embeddings and cross-encoder reranking, then uses a constrained LLM to score options with structured signals, applying deterministic rules for predictions.

In practice

• Implement dense embeddings for initial evidence retrieval.
• Use cross-encoders to refine evidence packs for LLMs.
• Detect "none of the above" options with lexical cues.

Topics

• Abductive Reasoning
• Large Language Models
• Information Retrieval
• SemEval-2026 Task 12
• Evidence-Guided Scoring
• Dense Embeddings

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Abductive Reasoning with Probabilistic Commonsense — PACS enhances LLM abductive reasoning by probabilistically modeling diverse commonsense beliefs, outperforming prior methods.
• Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement — Decomposing LLM evaluation into binary questions provides interpretable, multi-dimensional scores and enables self-improvement.
• Zero-Shot Goal Recognition with Large Language Models — LLM goal recognition competence varies significantly, with some models integrating evidence effectively while others rely on world-knowledge priors.
• When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering — LLMs require retrieval augmentation and strong reasoning for reliable off-guideline medical question answering.
• AILS-NTUA at SemEval-2026 Task 12: Graph-Based Retrieval and Reflective Prompting for Abductive Event Reasoning — AILS-NTUA's three-stage system won SemEval 2026 Task 12 for Abductive Event Reasoning, achieving a first-place rank with an accuracy score of 0.95.
• Truncated Step-Level Sampling with Process Rewards for Retrieval-Augmented Reasoning — Truncated step-level sampling with dense LLM-as-judge rewards significantly improves RL for search-augmented LLM reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.