From Isolated Tasks to Structured Capabilities: A Multilayer Taxonomy for Large Language Models

2026-07-24 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new multilayer taxonomy for Large Language Model (LLM) evaluation has been introduced, organizing capabilities into 14 domains and 91 subskills across Primitive, Constructed, and Integrative layers. This taxonomy, guided by human cognitive science, addresses the current fragmentation of LLM evaluation, which is task-based rather than capability-focused. To demonstrate its utility, researchers screened 31,505 papers from ACL, AAAI, ICML, and NeurIPS between 2023 and 2025, mapping 15,934 LLM-focused papers. Analysis revealed research concentration on Language-Semantic Competence (22.3%), Reasoning (21.3%), Planning and Decision-Making (13.5%), and Perception (12.3%), while six domains appeared in fewer than 2% of papers. The taxonomy aims to improve research organization, coverage audits, and evaluation interpretation.

Key takeaway

For research scientists evaluating Large Language Models, this capability-based taxonomy offers a structured approach to move beyond isolated task performance. You should consider adopting this multilayer framework to organize your research, audit evaluation coverage, and formulate more precise hypotheses for LLM diagnosis and training. This shift can enhance cross-study comparisons and reveal critical capability gaps in current models.

Key insights

A new LLM taxonomy organizes evaluation by capabilities, not tasks, improving research and identifying gaps.

Principles

• LLM evaluation benefits from capability-centric organization.
• Human cognitive science can guide LLM capability definition.
• Developmental precedence informs capability layer assignments.

Method

The method involves defining 14 capability domains and 91 subskills across three layers, guided by human cognitive science, then mapping research papers to these capabilities via multi-model annotation.

In practice

• Use capability taxonomy for LLM evaluation design.
• Audit research coverage using capability domains.
• Formulate testable hypotheses for LLM diagnosis.

Topics

• LLM Evaluation
• Capability Taxonomy
• Cognitive Science
• Research Mapping
• Language-Semantic Competence
• Reasoning

Best for: AI Scientist, Research Scientist

Related on AIssential

• From Isolated Tasks to Structured Capabilities: A Multilayer Taxonomy for Large Language Models — A multi-layer taxonomy organizes LLM capabilities, moving beyond task-centric evaluation to reveal underlying cognitive structures and research gaps.
• Know More, Know Clearer: A Meta-Cognitive Framework for Knowledge Augmentation in Large Language Models — A meta-cognitive framework improves LLM knowledge augmentation by addressing confidence gaps and differentiating knowledge states.
• Metacognition in LLMs: Foundations, Progress, and Opportunities — Metacognition is a critical, foundational component of intelligence for capable, transparent AI systems, now being explored in LLMs.
• Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature — A multi-LLM council with an expert-guided ontology can quantitatively map scientific literature's support for hypotheses, revealing structured agreement and disagreement.
• Scientific exploration, collaboration and labor division in the large language model era — The LLM era correlates with a reorganization of scientific exploration, collaboration, and labor division.
• Integrating knowledge graphs and multilingual scholarly corpora for domain-adaptive LLMs in SSH — Adapting LLMs for SSH research requires integrating knowledge graphs, multilingual corpora, and robust evaluation within ethical frameworks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.