Are LLMs Bad at Moral Reasoning?

2026-06-10 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Ethics & Responsible AI · Depth: Expert, quick

Summary

A new analysis challenges previous pessimistic conclusions regarding large language models' (LLMs) moral reasoning capabilities, particularly those drawn from the MoReBench dataset. Earlier research benchmarked frontier AI models against 1,000 gold-standard human-authored rubrics for moral reasoning across various cases, yielding underwhelming results. This paper argues that if LLMs are tasked with generating scoring rubrics for moral analysis, rather than open-ended responses, their performance appears significantly more capable. The LLM-generated rubrics demonstrate better calibration to human-authored rubrics. Where discrepancies exist, they are attributed to the vast dimensionality of moral problems or human departures from rubric creation guidelines, suggesting LLMs possess greater moral reasoning capacity than initially believed.

Key takeaway

For AI Scientists and Ethicists designing or interpreting moral reasoning benchmarks, you should critically re-evaluate the task given to large language models. If your current evaluations rely on scoring LLM open-ended responses, consider shifting to a rubric generation task. This approach may reveal a significantly higher moral competence in LLMs, suggesting that current pessimistic conclusions might stem from methodological choices rather than inherent AI limitations. Adjusting your evaluation framework could lead to more accurate assessments.

Key insights

Re-tasking LLMs to generate moral reasoning rubrics reveals significantly higher moral competence than prior evaluations.

Principles

• Moral competence involves identifying and responding to moral reasons.
• Evaluation task design critically influences perceived AI capabilities.
• Moral problems often possess vast dimensionality.

Method

The method involves giving LLMs the task of generating scoring rubrics for moral analysis of cases, then comparing these generated rubrics against human-authored gold standards.

In practice

• Re-evaluate AI systems by altering the task design.
• Analyze discrepancies in moral reasoning as dimensionality issues.

Topics

• Large Language Models
• Moral Reasoning
• AI Ethics
• MoReBench Dataset
• Benchmark Design
• AI Evaluation

Best for: Research Scientist, AI Scientist, AI Ethicist

Related on AIssential

• Google DeepMind wants to know if chatbots are just virtue signaling — LLMs' apparent moral competence may be superficial, requiring rigorous evaluation beyond surface-level performance.
• Humans vs. Gary Marcus vs. Slate Star Codex: When is an AI failure actually a failure? — Human evaluation reveals GPT-3's "mistakes" often mirror human creativity or narrative, challenging strict logical reasoning benchmarks.
• Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment — The UPBench framework reveals LLMs struggle with context-sensitive, lower-order urban planning tasks despite performing well on higher-order analysis.
• Cognitive Effects and Biases in Large Language Models — This tutorial clarifies cognitive effects and biases in LLMs by bridging psychology and NLP.
• Think Before You Lie: How Reasoning Improves Honesty — LLM reasoning increases honesty by nudging models toward stable, honest defaults in their representational space.
• BREAKING: LLM “reasoning” continues to be deeply flawed — LLMs, even those marketed for reasoning, continue to exhibit fundamental and widespread reasoning failures.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.