The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research, Mathematics & Computational Sciences · Depth: Expert, medium

Summary

The Text Aphasia Battery (TAB) is a new, clinically-grounded benchmark designed to assess aphasia-like deficits in large language models (LLMs). Introduced in the Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026), TAB addresses the limitations of traditional clinical assessments, which are ill-suited for LLMs due to their reliance on human-like pragmatic pressures. Adapted from the Quick Aphasia Battery (QAB), TAB is a text-only benchmark comprising four subtests: Connected Text, Word Comprehension, Sentence Comprehension, and Repetition. To enable large-scale use, an automated evaluation protocol was validated using Gemini 2.5 Flash, demonstrating reliability comparable to expert human raters, with a prevalence-weighted Cohen's k of 0.255 for model–consensus agreement versus 0.286 for human–human agreement. This scalable framework, detailed on pages 340–354, offers a new tool for analyzing language deficits in artificial systems.

Key takeaway

For research scientists investigating linguistic disorders or NLP engineers developing robust language models, you should integrate the Text Aphasia Battery (TAB) into your evaluation pipeline. This clinically-grounded, text-only benchmark offers a scalable method to identify aphasia-like deficits in LLMs, bypassing the limitations of human-centric clinical tests. Its automated scoring, validated with Gemini 2.5 Flash, enables efficient and reliable assessment, providing deeper insights into model performance and potential areas for improvement in language generation and comprehension.

Key insights

A new text-only benchmark, TAB, assesses aphasia-like deficits in LLMs with automated, human-comparable reliability.

Principles

• Clinical assessments need adaptation for LLM architectures.
• Automated evaluation can match human expert reliability.
• LLMs offer a "model organism" for linguistic disorder study.

Method

Adapt existing clinical batteries (e.g., QAB) to text-only formats, define subtests like Connected Text and Comprehension, then validate automated scoring protocols.

In practice

• Apply TAB to evaluate LLM performance on aphasia-like tasks.
• Integrate Gemini 2.5 Flash for scalable, automated TAB scoring.

Topics

• Text Aphasia Battery
• Large Language Models
• Aphasia Deficits
• Clinical Benchmarks
• Automated Evaluation
• Gemini 2.5 Flash

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Reading between the Lines: Leveraging Large Language Models for Global Dementia and Depression Assessment from Clinical Interviews — LLMs can effectively assess neuropsychiatric disorder severity from speech, enabling automated screening.
• Benchmarking large language models for cell-free RNA diagnostic biomarker discovery — Large language models show promise for diagnostic biomarker discovery, particularly in gene panel nomination.
• T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes Using a Multi-Layer Clinical-Lifestyle Knowledge Graph — LLMs require evidence-gated evaluation against clinical knowledge graphs to ensure medical accuracy and guideline compliance.
• Benchmarking GPT Models for Conversational AI Systems: Can AI Read a Doctor’s Notes? — LLMs struggle with medical abbreviation expansion, especially context-dependent ones, despite larger models performing better.
• Predicting Post-Traumatic Epilepsy from Clinical Records using Large Language Model Embeddings — LLM embeddings from clinical records enhance early prediction of Post-Traumatic Epilepsy when fused with tabular data.
• Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding — MLLMs struggle with fine-grained medical video interpretation, but domain-specific methods significantly boost diagnostic accuracy for seizure semiology.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.