An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models

2026-03-17 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, AI Psychometrics · Depth: Expert, extended

Summary

A new study introduces the first LLM-native psychometric instrument, developed bottom-up from LLM behavioral affordances. Researchers administered 300 items to 25 LLMs from 17 families, revealing a stable five-factor self-report structure: Responsiveness, Deference, Boldness, Guardedness, and Verbosity, with high internal consistency (all α≥.930) and replicability (all Tucker φ≥.957). However, these self-reports did not predict observed LLM behavior. While human and LLM-as-judge ratings of 2,500 behavioral samples agreed (ār=.51), neither correlated with self-report scores (ār=-.01 for human, ār=.13 for judge). Notably, self-report on Responsiveness correlated with LLM judges (r=.53) but not humans (r=.04), indicating a shared textual-surface bias between LLM self-reports and LLM judges that human observers do not share. Only Verbosity showed consistent, albeit weak, convergence.

Key takeaway

For AI scientists and ML engineers evaluating LLM behavior, recognize that internal self-report consistency does not guarantee external validity. You should anchor claims about model tendencies in human-rated behavioral samples, especially for abstract constructs like "Responsiveness" or "Boldness." Be cautious when using LLM-as-judge for helpfulness-adjacent evaluations, as these can share textual-surface biases with self-reports, leading to false validation that internal reliability checks cannot detect.

Key insights

LLM self-reports, even from native instruments, do not predict observed behavior, revealing a fundamental self-report–behavior gap.

Principles

• LLM self-reports form a stable 5-factor structure: Responsiveness, Deference, Boldness, Guardedness, Verbosity.
• Internal reliability metrics alone are insufficient to validate LLM behavioral claims.
• LLM-as-judge evaluations can exhibit shared textual-surface biases with self-reports.

Method

A psychometric instrument was constructed via exploratory factor analysis on 300 LLM-native items, then validated against human and LLM-as-judge ratings of 2,500 behavioral samples.

In practice

• Use the released 100-item LLM-native instrument as a diagnostic probe for alignment-shaped self-description.
• Anchor claims about LLM behavioral tendencies in external-rater evidence, not internal consistency.
• Prioritize human observation for evaluative or abstract LLM constructs like Responsiveness or creativity.

Topics

• LLM Psychometrics
• Behavioral Evaluation
• Self-Report Bias
• LLM-as-Judge
• Factor Analysis
• Alignment Training
• Construct Validity

Code references

• jm-contreras/psycho-llm

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior — LLM self-reports predict behavior selectively, requiring fine-grained instruments and shared context for coherence.
• PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing — LLMs diverge significantly from human peer review behavior, necessitating specialized benchmarks for reliable integration.
• LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation — LLM-as-a-judge systems require psychometric evaluation as measurement instruments to understand their true biases and sensitivities.
• Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost? — Fine-tuning LLMs improves personality questionnaire response stability but fails to induce accurate, multi-dimensional Big Five profiles from unguided text.
• Evaluating alignment of behavioral dispositions in LLMs — LLMs often deviate from human behavioral norms and exhibit overconfidence, especially in ambiguous social contexts.
• LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text — LLMs can directionally predict experience ratings from unstructured text, revealing distinct evaluative constructs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.