XDomainBench: Diagnosing Reasoning Collapse in High-Dimensional Scientific Knowledge Composition

2026-05-14 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

XDomainBench is a new diagnostic benchmark designed to evaluate Large Language Models' (LLMs) capacity for compositional generalization in scientific knowledge synthesis, particularly in interactive, interdisciplinary scenarios. It formalizes composition order and mixture structure to systematically stress-test LLMs from single-discipline to inter-disciplinary tasks. The benchmark includes 8,598 interactive sessions across 20 domains and 4 task categories, featuring 8 realistic trajectory patterns that simulate real AI4S (AI for Science) scenarios, covering varying difficulty and domain-mixture dynamics. Initial large-scale evaluations using XDomainBench reveal a systematic reasoning collapse in LLMs as the composition order increases. This collapse is attributed to two main factors: direct difficulty increases from domain composition and indirect interaction-amplified failures, which lead to error accumulation, reasoning breaks, and domain confusion, ultimately causing session collapse.

Key takeaway

For AI Scientists developing LLMs for scientific applications, you should prioritize improving compositional generalization, especially in interactive, multi-domain contexts. The XDomainBench findings indicate that current LLMs struggle with error accumulation and domain confusion as task complexity increases. Focus on architectural or training innovations that mitigate these "reasoning collapse" factors to enhance real-world scientific utility.

Key insights

LLMs exhibit systematic reasoning collapse in complex, interactive scientific knowledge composition tasks.

Principles

• Compositional generalization is critical for scientific knowledge synthesis.
• Interactive workflows expose LLM capability boundaries.

Method

XDomainBench formalizes composition order and mixture structure, using 8,598 interactive sessions across 20 domains and 4 task categories with 8 trajectory patterns to diagnose LLM reasoning collapse.

In practice

• Evaluate LLMs on interdisciplinary reasoning tasks.
• Stress-test models with increasing composition order.

Topics

• XDomainBench
• Large Language Models
• Scientific Knowledge Composition
• Reasoning Collapse
• Interdisciplinary Reasoning

Best for: AI Scientist, Research Scientist

Related on AIssential

• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory — LLMs struggle with mathematical reasoning, particularly graduate-level proofs, with GPT-5 significantly outperforming other models.
• Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference — MACR resolves LLM knowledge conflicts by explicitly assessing confidence in sources and employing multi-agent reasoning, moving beyond binary reliability assumptions.
• DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning — DecompSR is a provably correct dataset for fine-grained analysis of LLM compositional spatial reasoning.
• To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning — Transformer models exhibit representational collapse of unseen token unembeddings, hindering symbolic reasoning generalization.
• Correct Chains, Wrong Answers: Dissociating Reasoning from Output in LLM Logic — LLMs can produce correct reasoning steps but still output wrong final answers, revealing a reasoning-output dissociation.
• Written by AI, Managed by AI: Semantic Space Control and Index Sickness Elimination Across 391 Consecutive Sessions — Overly complex symbolic systems cause LLMs to lose semantic grounding, necessitating natural language-based separation for robust long-horizon collaboration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.