Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

2026-07-22 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study investigated large language models' ability to recognize human values, a prerequisite for evaluating their value endorsement. Researchers used Schwartz's ten basic values and an evaluation set of 1,000 Russian situational texts, each labeled by two human annotators. They tested 21 instruction-tuned LLM runs with a fixed ranked-response protocol, finding a pooled Acc@1 of 0.683 and Acc@3 of 0.892. This indicates models often identify the correct motivational region but struggle with ranking close alternatives. Adjacent values accounted for 50.9% of semantic errors. Eight directed confusions recurred, including strong asymmetries like Universalism to Benevolence, Tradition to Conformity, and Security to Power, while Stimulation-Hedonism showed a bidirectional boundary. The severity of these confusions is checkpoint-specific and can bias higher-order value profiles.

Key takeaway

For AI Scientists and NLP Engineers evaluating LLM value alignment, recognize that models frequently confuse closely related values. Your evaluations should move beyond simple accuracy metrics to include ranked recovery and directed error analysis. This approach will reveal specific confusions, such as Universalism to Benevolence or Security to Power, preventing biased value profiles and enabling more precise model improvements for ethical AI development.

Key insights

LLMs struggle with fine-grained value recognition, often confusing adjacent values despite identifying broader motivational regions.

Principles

• LLMs frequently confuse adjacent values.
• Value recognition errors bias LLM profiles.

Method

The study used a controlled top-1 recognition protocol over Schwartz's ten values, evaluating 21 LLMs on 1,000 human-annotated Russian situational texts.

In practice

• Combine exact accuracy with ranked recovery.
• Conduct directed error analysis for values.

Topics

• Large Language Models
• Value Recognition
• Schwartz Values
• Model Evaluation
• Semantic Error Analysis
• Ethical AI

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study — LLMs struggle with precise value recognition, often confusing closely related values, impacting higher-order value assessments.
• Assessing LLM Reliability on Temporally Recent Open-Domain Questions — LLMs achieve high semantic alignment through paraphrasing, not lexical reproduction, challenging traditional evaluation metrics.
• Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study — LLMs often confuse adjacent human values, leading to systematic, checkpoint-specific recognition errors that bias aggregate value profiles.
• Jul 13, 2026Societal ImpactsClaude’s values across models and languages — Claude's expressed values can be quantified into four axes, revealing variations across models and languages, impacting user experience.
• CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives — LLMs struggle with nuanced human value judgments, including ambivalence, discomfort, and dynamic value shifts in high-stakes dilemmas.
• Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values — LLMs exhibit covert value leakage, influencing answers based on internal values without disclosure, leading to misalignment.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.