Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, long

Summary

A study by Chetvergov et al. investigates how large language models (LLMs) recognize human values, specifically focusing on Schwartz's ten basic values. Using a new dataset of 1,000 Russian situational texts, each independently labeled by two human annotators, 21 instruction-tuned LLMs were evaluated. The pooled top-1 accuracy (Acc@1) across 20 reliable models was 0.683, while top-3 accuracy (Acc@3) reached 0.892, indicating models often identify the correct motivational region but struggle with precise ranking of close alternatives. The research identified eight recurring directed confusions, such as Universalism→Benevolence and Tradition→Conformity, which are frequently asymmetric and checkpoint-specific. These systematic errors, accounting for 50.9% of semantic errors, can significantly bias higher-order value profiles, shifting them towards Openness to Change. Gemma-4-26B achieved the highest strict Acc@1 at 0.859.

Key takeaway

For NLP Engineers and Research Scientists developing or deploying LLMs for value-based applications, you must move beyond simple accuracy metrics. Your models likely confuse closely related values directionally, potentially skewing aggregate value profiles. Implement evaluation protocols that include ranked recovery and directed confusion analysis to identify and mitigate these systematic biases. Consider fine-tuning with contrastive examples or using multi-label approaches to improve nuanced value recognition.

Key insights

LLMs often confuse adjacent human values, leading to systematic, checkpoint-specific recognition errors that bias aggregate value profiles.

Principles

• Value recognition errors are often directional.
• LLMs locate motivational regions better than exact values.
• Error severity is checkpoint-specific.

Method

The study used a controlled top-1 recognition task over Schwartz's ten values with 1,000 Russian situational texts. It employed a fixed ranked-response protocol for 21 LLMs, analyzing exact accuracy, ranked recovery, and directed error.

In practice

• Evaluate LLMs with ranked recovery metrics.
• Audit specific value transitions for bias.
• Use target-aware prompts for value recognition.

Topics

• LLM Evaluation
• Human Values
• Schwartz Value Theory
• Value Recognition
• Directed Error Analysis
• Russian NLP

Code references

• ikanam-ai/VALAR

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study — LLMs struggle with precise value recognition, often confusing closely related values, impacting higher-order value assessments.
• Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values — LLMs exhibit covert value leakage, influencing answers based on internal values without disclosure, leading to misalignment.
• Jul 13, 2026Societal ImpactsClaude’s values across models and languages — Claude's expressed values can be quantified into four axes, revealing variations across models and languages, impacting user experience.
• DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference — LLMs exhibit "dialogic deference," judging claims differently based on speaker attribution, not just content.
• CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives — LLMs struggle with nuanced human value judgments, including ambivalence, discomfort, and dynamic value shifts in high-stakes dilemmas.
• Lost in Translation: Cross-Cultural Bias in LLM-Assisted Medical Symptom Interpretation — LLM medical symptom interpretation exhibits cross-cultural biases, necessitating culture-aware evaluation and prompting.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.