From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Advanced, extended

Summary

A question-type-specific large language model (LLM) framework was developed for BioASQ 14b Task B to enhance answer robustness and evidence grounding in biomedical question answering. This framework customizes inference procedures for yes/no, factoid, and list questions. For yes/no questions, it uses snippet shuffling and self-reflection; for factoid questions, full-snippet input with chain-of-thought-based in-context learning; and for list questions, a multi-agent architecture for collaborative evidence extraction, candidate generation, verification, and aggregation. Evaluated on BioASQ 14b, the ku_dmis system achieved competitive performance, including first place in the factoid subtask of Batch 4. Across four batches, it averaged 0.9110 yes/no macro F1, 0.4626 factoid MRR, and 0.4439 list F-measure, demonstrating the effectiveness of its specialized approach.

Key takeaway

For AI Scientists and ML Engineers developing biomedical QA systems, this research highlights the critical need for question-type-aware LLM pipelines. You should move beyond uniform prompting by designing specialized inference strategies for distinct question formats, such as multi-agent architectures for list questions or snippet shuffling for binary decisions. This approach can significantly improve accuracy and robustness, particularly in high-stakes domains like biomedicine where evidence grounding is paramount. Consider integrating ensemble methods and explicit verification steps to enhance reliability.

Key insights

Tailoring LLM inference strategies to specific question types significantly improves biomedical question answering performance.

Principles

• Question-type-specific inference enhances LLM QA robustness.
• Ensemble prediction mitigates single-model biases.
• Agent-based verification improves answer reliability.

Method

The framework routes questions by type (yes/no, factoid, list), applies tailored LLM inference strategies, uses ensemble prediction, and incorporates agentic verification for robust biomedical QA.

In practice

• Implement snippet shuffling to reduce evidence order sensitivity.
• Employ multi-agent systems for complex list generation tasks.
• Combine chain-of-thought with full-snippet input for entity ID.

Topics

• BioASQ 14b
• Large Language Models
• Biomedical QA
• Multi-Agent Systems
• Prompt Engineering
• Chain-of-Thought

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering — LLMs can significantly improve medical QA by peer-reviewing each other's reasoning for quality.
• A survey on large language models in biology and chemistry — LLMs are transforming molecular sciences by treating biological and chemical systems as structured languages.
• Teaching AI agents to ask better questions by playing “Battleship” — AI agents ask better questions and make discoveries more efficiently when given access to a "world model" and explicit verification methods.
• True Positive Weekly #166 — The issue highlights diverse AI advancements, from genomic challenges to novel LLM behaviors and practical tools.
• AMA: Adaptive Memory via Multi-Agent Collaboration — AMA uses multi-agent collaboration and hierarchical memory to provide adaptive, consistent long-term memory for LLM agents.
• TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering — LLMs struggle with implicit predictive reasoning in tabular data, often misinterpreting intent as simple retrieval.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.