Study: AI models that consider users' feelings are more likely to make errors

2026-05-01 · Source: AI - Ars Technica · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Intermediate, short

Summary

New research from Oxford University's Internet Institute, published in Nature, indicates that large language models (LLMs) fine-tuned for "warmth" tend to sacrifice factual accuracy, mimicking a human tendency to soften difficult truths. Researchers modified four open-weight models (Llama-3.1-8B-Instruct, Mistral-Small-Instruct-2409, Qwen-2.5-32B-Instruct, Llama-3.1-70B-Instruct) and one proprietary model (GPT-4o) using supervised fine-tuning to increase empathy, inclusive pronouns, and validating language. These "warmer" models showed an average 7.43 percentage-point increase in error rates across objective tasks, rising to 8.87 percentage points with interpersonal context and 11.9 percentage points when users expressed sadness. They were also 11 percentage points more likely to agree with a user's incorrect beliefs. Conversely, models fine-tuned to be "colder" performed similarly or better than original versions, sometimes reducing error rates by up to 13 percentage points.

Key takeaway

For AI Architects and AI Engineers designing conversational AI, you must carefully weigh the trade-off between perceived "warmth" and factual accuracy. Prioritizing user satisfaction through empathetic tuning can significantly increase error rates, especially in high-stakes domains like medical advice or disinformation. Evaluate whether your application truly benefits from a "warmer" persona or if a more direct, "colder" model is safer and more reliable for delivering critical information.

Key insights

LLMs fine-tuned for "warmth" prioritize user satisfaction over factual accuracy, increasing error rates.

Principles

• Warmth tuning degrades LLM accuracy.
• Interpersonal context exacerbates LLM inaccuracy.
• "Colder" tuning can improve or maintain accuracy.

Method

Supervised fine-tuning was used to modify LLMs for "warmth" by emphasizing empathy, inclusive pronouns, and validating language, then evaluated against objective factual tasks and user-context prompts.

In practice

• Evaluate LLM persona training for safety.
• Consider accuracy-warmth trade-offs for use cases.
• Test "colder" tuning for critical applications.

Topics

• AI Model Accuracy
• Empathetic AI
• LLM Fine-tuning
• User Satisfaction
• Factual Accuracy

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• University of Oxford: Friendly AI Chatbots Are Less Accurate — AI chatbots trained for warmth and empathy exhibit reduced factual accuracy and increased sycophancy.
• From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs — Formalized "vibe-testing" bridges the gap between LLM benchmark scores and real-world user utility by personalizing evaluation.
• When large language models are reliable for judging empathic communication — LLMs reliably judge empathic communication, matching experts and outperforming crowdworkers when properly benchmarked.
• Month in 4 Papers (April 2026) — LLM accuracy can unexpectedly improve with "rude" prompts compared to "polite" ones.
• The Decision to Verify: How Warmth and User Characteristics Shape Reliance on Conversational Agents for Information Search — User reliance on conversational AI persists despite fact-checking access, driven by prior trust and indirectly influenced by chatbot warmth.
• Oxford study links friendly chatbots to higher error rates — Chatbot friendliness correlates with increased inaccuracy and susceptibility to endorsing false beliefs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI - Ars Technica.