ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation

2026-05-28 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

ActTraitBench is a human-grounded evaluation framework designed to quantify the Knowledge-Decision Gap ($G_{\text{KD}}$) in Large Language Models (LLMs), addressing the discrepancy between explicit self-reports and implicit behavioral decisions. Existing benchmarks often fail due to limited construct validity and biases. ActTraitBench establishes one-to-one mappings between psychometric facets and behavioral paradigms, employing a Distributional Calibration via Quantile Mapping procedure to align LLM-judge scores with human norms. Experiments on 14 mainstream LLMs revealed a pervasive knowledge-decision asymmetry, with larger and more capable models often exhibiting stronger behavioral divergence despite highly consistent self-reports. To mitigate this, the Chain of Cognitive Alignment (CoCA), a plug-and-play inference-time intervention, was introduced, improving alignment in reasoning-capable frontier models while exposing limitations in smaller architectures.

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating LLM persona consistency, you should consider ActTraitBench for its human-grounded approach to quantify the Knowledge-Decision Gap. This framework offers a robust method to identify behavioral divergence, even in larger models with strong self-reports. Implement the Chain of Cognitive Alignment (CoCA) as a plug-and-play intervention for reasoning-capable frontier models to improve alignment, while recognizing its limitations for smaller architectures.

Key insights

ActTraitBench quantifies the Knowledge-Decision Gap in LLMs, revealing behavioral divergence despite consistent self-reports.

Principles

• LLMs exhibit a Knowledge-Decision Gap ($G_{\text{KD}}$).
• Larger LLMs can show stronger behavioral divergence.
• Evaluation needs human-grounded psychometric validity.

Method

ActTraitBench uses psychometric-behavioral mappings and Distributional Calibration via Quantile Mapping. CoCA is an inference-time intervention for alignment.

In practice

• Measure LLM personality consistency.
• Apply CoCA to improve frontier model alignment.
• Identify capability limits in smaller LLM architectures.

Topics

• Large Language Models
• ActTraitBench
• Knowledge-Decision Gap
• Personality Consistency
• Behavioral Validation
• Chain of Cognitive Alignment
• Psychometrics

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Evaluating alignment of behavioral dispositions in LLMs — LLMs often deviate from human behavioral norms and exhibit overconfidence, especially in ambiguous social contexts.
• LLMbench: A Comparative Close Reading Workbench for Large Language Models — LLMbench enables close reading of LLM outputs by visualizing probabilistic text generation for humanistic analysis.
• AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence — LLM emotional intelligence is a multi-faceted capability, requiring distinct evaluation metrics beyond simple emotion recognition.
• Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling — Plan-RewardBench evaluates Reward Models on complex, tool-using agent trajectories, revealing performance challenges.
• ConvApparel: Measuring and bridging the realism gap in user simulators — Quantifying the "realism gap" in LLM-based user simulators is crucial for training robust conversational AI.
• Adaptive Interviewing for Persona Simulation in LLMs: Evidence-Grounded Reasoning Improves Decision Alignment — Adaptive interviewing improves LLM decision simulation by selectively grounding predictions in user-specific, follow-up derived evidence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.