Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

2026-07-22 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study investigated large language models' (LLMs) ability to recognize human values, a prerequisite for value-based evaluations. Researchers evaluated 21 instruction-tuned LLM runs using a fixed ranked-response protocol on 1,000 Russian situational texts, balanced across Schwartz's ten basic values and human-annotated. The pooled Acc@1 was 0.683, with Acc@3 at 0.892, indicating models often identify the correct motivational region but struggle with ranking close alternatives. Semantic errors frequently involved adjacent values (50.9%), significantly higher than a checkpoint-specific null (24.4%). Eight directed confusions, such as Universalism to Benevolence and Security to Power, recurred across models, with severity varying by checkpoint and potentially biasing higher-order value profiles.

Key takeaway

For AI ethicists and NLP engineers developing or evaluating LLMs for value alignment, you must move beyond simple accuracy metrics. Your evaluations should incorporate ranked recovery and directed error analysis to uncover specific value confusions. Understanding these directed biases, like Universalism to Benevolence or Security to Power, is crucial, as their checkpoint-specific severity can significantly distort higher-order value profiles and lead to misinterpretations of model behavior.

Key insights

LLMs struggle with precise value recognition, often confusing closely related values, impacting higher-order value assessments.

Principles

• LLMs frequently locate correct motivational regions.
• Adjacent values account for most semantic errors.
• Directed confusions are often asymmetric.

Method

The study used a controlled top-1 recognition task over Schwartz's ten basic values, evaluating 21 LLMs on 1,000 human-annotated Russian situational texts via a fixed ranked-response protocol.

In practice

• Combine exact accuracy with ranked recovery.
• Perform directed error analysis for value recognition.
• Consider Schwartz's ten basic values for evaluation.

Topics

• Large Language Models
• Value Alignment
• Schwartz's Theory of Basic Values
• Semantic Error Analysis
• LLM Evaluation
• Natural Language Processing

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study — LLMs struggle with fine-grained value recognition, often confusing adjacent values despite identifying broader motivational regions.
• A Unified Moral-Value Dataset for Instruction Tuning — A unified moral-value dataset and instruction tuning pipeline improves LLM value alignment while preserving general task performance.
• The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs — Current LLM alignment can overgeneralize safety cues, overriding explicit evidence in stereotype-sensitive contexts.
• Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations — LLM failures in systematic review screening are rooted in identifiable semantic and inference issues, not merely quantitative disagreement.
• CrosSing: Cross-Scale Reasoning Evaluation on LLMs against Humans — LLMs struggle with scalar reasoning in overinformative contexts where humans excel, revealing a fundamental difference.
• More Context, Larger Models, or Moral Knowledge? A Systematic Study of Schwartz Value Detection in Political Texts — Value detection performance depends critically on context, external knowledge, and model architecture, not just scale.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.