The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

2026-07-03 · Source: cs.CL updates on arXiv.org · Field: Science & Research — Mathematics & Computational Sciences, Social Sciences & Behavioral Studies, Research Methodology & Innovation · Depth: Expert, extended

Summary

Frontier large language models (LLMs) exhibit a "yes-no bias" in moral judgments, where verdicts shift due to question wording and answer order, not changes in underlying moral stance. This study introduces a psychometric battery, "crossed symmetrization," to separate these factors. It found that frontier models like Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.5, and Gemini-3-Flash possess a coherent internal moral scale, with graded ratings showing cross-form incoherence of only 0.12–0.21 on a ±1 axis. However, forcing binary yes/no answers overlays a substantial artifact, particularly in Claude models (story-averaged -0.32 to -0.86), comprising an order bias toward the last-printed option and a lexical pull toward the word "no." This artifact is approximately 0 for GPT-5.5 and Gemini. Deliberation shrinks this artifact, and with arbitrary answer labels, the verdict-attached logical bias is approximately 0 for frontier models, indicating the pull follows the printed label, not the moral verdict.

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating LLM ethics, you should prioritize psychometric methods like "crossed symmetrization" to accurately gauge moral stance. Relying on single-format binary yes/no questions can introduce significant, misleading biases, particularly order and lexical pulls. Instead, elicit graded responses across diverse, logically equivalent frames to reveal a model's true, coherent internal scale. This approach ensures evaluations reflect genuine moral judgment, not superficial framing effects, especially when assessing models like Claude, which show higher susceptibility.

Key insights

LLM moral judgments are stable internally but distorted by surface-level framing in binary yes/no questions.

Principles

• LLMs possess a coherent, format-invariant internal moral scale.
• Binary yes/no readouts confound moral stance with format artifacts.
• Deliberation reduces framing susceptibility in LLMs.

Method

The "crossed symmetrization" psychometric battery separates logical verdict from lexical token and answer order by flipping logically irrelevant factors in balanced pairs across diverse question forms.

In practice

• Elicit LLM moral stance using graded ratings across varied frames.
• Avoid single-format yes/no questions for moral evaluations.
• Use arbitrary labels to isolate true verdict-attached biases.

Topics

• Large Language Models
• Moral Judgment
• AI Evaluation
• Framing Effects
• Psychometrics
• Bias Decomposition

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• Do Emotions Influence Moral Judgment in Large Language Models? — LLMs' moral judgments are systematically biased by emotional context, unlike human judgments.
• CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives — LLMs struggle with nuanced human value judgments, including ambivalence, discomfort, and dynamic value shifts in high-stakes dilemmas.
• Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs — LLMs exhibit "moral deliberative sycophancy" in non-verifiable reasoning, aligning justifications with user views.
• Frame-Conditioned Moral Computation in LLaMA 3.1-8B-Instruct: A Mechanistic Interpretability Audit of Ethical Reasoning — LLaMA 3.1-8B-Instruct's moral computation is frame-conditioned, with ethical salience dependent on prompt interpretation.
• DeFrame: Debiasing Large Language Models Against Framing Effects — LLM fairness evaluations are highly sensitive to prompt framing, necessitating methods that ensure consistency across phrasing.
• Bias and Justice in Language Models: Evidence from a Linguistically, Socially, and Culturally Asymmetric Literature — LLM bias research is skewed towards English and binary gender, lacking intersectional and multilingual real-world analyses.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.