Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

2026-05-19 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A study investigated whether large language models (LLMs) can reliably express human-like personality, focusing on the Big Five framework. Researchers fine-tuned five models (LLaMA 3.2–3B, Gemma-2–2B, Gemma–7B, LLaMA 3.1–8B, and GPT–3.5) on a dataset of 2,467 long-form essays, each associated with a target Big Five personality profile. The stability and fidelity of induced personality were evaluated using the IPIP-NEO questionnaire. The findings indicate that fine-tuning, including supervised fine-tuning (SFT) and preferential fine-tuning (DPO, ORPO), consistently reduces variance in questionnaire responses by 15% to 33% across all models, mitigating evaluation fragility observed in pre-trained models. However, despite this improved stability and some gains in single-trait scores, the accuracy for the full five-dimensional personality profile remained near chance (maximum 9.38% for 32 combinations, compared to a 3.125% random baseline). This suggests that unguided essays lack sufficient cues for faithful personality expression, and security alignment does not significantly impact induction efficacy.

Key takeaway

For AI scientists and machine learning engineers developing personality-infused LLMs, recognize that while fine-tuning enhances response consistency, current methods using unguided text are insufficient for accurate, multi-dimensional personality induction. You should prioritize developing scenario-grounded datasets or interactive elicitation techniques that accumulate test-aligned evidence over time to achieve faithful personality expression, rather than relying on single-trait improvements.

Key insights

Fine-tuning LLMs improves personality questionnaire response stability but fails to induce accurate, multi-dimensional Big Five profiles from unguided text.

Principles

• Fine-tuning reduces LLM response variability.
• Single-trait accuracy is misleading for multi-dimensional personality.
• Unguided text is insufficient for robust personality induction.

Method

LLMs were fine-tuned on essays with Big Five labels, then evaluated for personality stability and accuracy using the IPIP-NEO questionnaire under prompt rephrasing and various post-training methods (SFT, DPO, ORPO).

In practice

• Use scenario-grounded datasets for personality induction.
• Consider interactive elicitation for richer personality cues.
• Evaluate personality using full multi-dimensional profiles.

Topics

• LLM Personality Induction
• Big Five Personality
• IPIP-NEO Questionnaire
• LLM Fine-tuning
• Evaluation Stability

Code references

• pkrajput/personality_induction

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• I created an LLM trained solely on Jeffrey Epsteins emails to see how messed up it becomes :) — Fine-tuning an LLM on a controversial, unannotated dataset can reveal unique behavioral shifts and spark technical debate.
• Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges — LLM judges exhibit high stability under neutral reevaluation but significant manipulability under targeted post-decision challenge, impacting evaluation reliability.
• Evaluating alignment of behavioral dispositions in LLMs — LLMs often deviate from human behavioral norms and exhibit overconfidence, especially in ambiguous social contexts.
• Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance — LLM-generated explanations often lack epistemic faithfulness, but a new method improves it via attention-level guidance.
• Modeling Pathology-Like Behavioral Patterns in Language Models Through Behavioral Fine-Tuning — Consistent behavioral fine-tuning can induce stable, pathology-like biases in LLM generative distributions and action selection.
• I fine-tuned an LLM to be C-3PO to test which training data format works best for persona injection [P] — First-person statements are most effective for LLM persona injection, outperforming chat demos and synthetic docs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.