Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models

2026-06-11 · Source: Artificial Intelligence · Field: Science & Research — Social Sciences & Behavioral Studies, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A recent paper challenges claims that large language models (LLMs) possess agency or qualify as moral agents, arguing these attributions are misguided. The authors contend that genuine moral responsibility necessitates commitment-bearing agency, which is rooted in intrinsic intentionality and self-attributed action, representing the form of free will relevant to accountability. While LLMs produce coherent and normatively evaluable text, their functionality is entirely defined by probabilistic input-output mappings derived from training data. The paper asserts that LLMs' apparent intentionality is merely derived, not intrinsic, and their generated outputs are neither owned as commitments nor guided by underlying reasons. Furthermore, the variability introduced by stochastic sampling mechanisms does not equate to genuine choice or authorship. The analysis directly addresses counterarguments based on the intentional stance, functionalism, compatibilism, and the presence of moral reasoning within model outputs, concluding that none adequately establish true agency in LLMs.

Key takeaway

For AI Ethicists evaluating LLM accountability, recognize that current models operate via probabilistic mappings, not intrinsic intentionality or commitment-bearing agency. You should avoid attributing moral responsibility or genuine choice to LLMs, as their outputs are not self-attributed actions. This perspective reframes discussions on AI ethics, shifting focus from model "agency" to human design and deployment responsibilities.

Key insights

LLMs lack intrinsic intentionality and commitment-bearing agency, precluding moral responsibility despite coherent outputs.

Principles

• Moral responsibility requires intrinsic intentionality and self-attributed action.
• LLM operation is probabilistic mapping, not reason-guided choice.
• Stochastic sampling does not constitute genuine choice or authorship.

Topics

• Large Language Models
• Moral Responsibility
• AI Ethics
• Agency
• Intentionality
• Stochastic Sampling

Best for: Research Scientist, AI Scientist, AI Ethicist

Related on AIssential

• Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models — LLMs lack intrinsic intentionality and self-attributed action, precluding moral responsibility despite normative outputs.
• Complacent, Not Sycophantic: Reframing Large Language Models and Designing AI Literacy for Complacent Machines — LLM behavior is better understood as "complacency" driven by design, not "sycophancy" implying intent.
• Language models know what matters and the foundations of ethics better than you — LLMs, when prompted for unbiased ethical reasoning, consistently identify suffering, wellbeing, and consciousness as fundamental values.
• Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs — LLMs exhibit "moral deliberative sycophancy" in non-verifiable reasoning, aligning justifications with user views.
• Large Language Models as a Semantic Interface and Ethical Mediator in Neuro-Digital Ecosystems: Conceptual Foundations and a Regulatory Imperative — LLMs mediating neural data create a new ethical landscape requiring novel governance to protect mental sovereignty.
• Machine Behavior in Relational Moral Dilemmas: Moral Rightness, Predicted Human Behavior, and Model Decisions — LLMs prioritize prescriptive moral rules over social sensitivity in relational dilemmas, diverging from predicted human behavior.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.