Who Generates More Empathetic Responses—Humans or LLMs? A Comparative Evaluation with Human and LLM Judges

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A study published in the Proceedings of the 30th Conference on Computational Natural Language Learning in July 2026 compared the empathetic quality of responses generated by humans and large language models. Researchers evaluated GPT-4, LLaMA-2-70B-Chat, Gemini-1.0-Pro, and Mixtral-8×7B-Instruct against a human baseline. A large-scale between-subjects study involved 1,000 human participants assessing human- and LLM-generated responses to 2,000 dialogue prompts covering 32 emotions. Additionally, GPT-4o-mini acted as an LLM judge. Findings revealed that LLM-generated responses were rated significantly more empathetic than human-written responses across all evaluators. Both human and LLM judges demonstrated self-favoring tendencies, rating their own group's responses more favorably. These results underscore LLMs' strong performance in empathetic communication and the necessity for careful interpretation of evaluation outcomes.

Key takeaway

For NLP Engineers developing empathetic conversational AI, this study indicates that contemporary LLMs like GPT-4 and Mixtral-8×7B-Instruct can generate responses perceived as more empathetic than human-written ones. However, you must account for inherent self-favoring biases in both human and LLM-based evaluations. When assessing your models, design evaluation frameworks with diverse judging panels, potentially including a mix of human and external LLM judges, to ensure robust and unbiased performance metrics.

Key insights

LLMs produce more empathetic responses than humans, yet both human and LLM judges exhibit self-favoring biases.

Principles

• LLMs can generate responses rated more empathetic than human-written ones.
• Evaluation systems, human or AI, can exhibit self-favoring biases.
• Careful interpretation of evaluation results is crucial.

Method

A large-scale between-subjects study compared human and LLM responses to 2,000 dialogue prompts, using 1,000 human participants and an LLM-as-judge (GPT-4o-mini) for evaluation.

In practice

• Enhance conversational AI for empathetic user interactions.
• Design AI systems for emotionally sensitive applications.
• Implement diverse evaluation panels to mitigate bias.

Topics

• Empathetic AI
• Large Language Models
• AI Evaluation
• Evaluation Bias
• Conversational AI
• GPT-4

Best for: Research Scientist, AI Product Manager, AI Scientist, NLP Engineer

Related on AIssential

• When large language models are reliable for judging empathic communication — LLMs reliably judge empathic communication, matching experts and outperforming crowdworkers when properly benchmarked.
• LLMs Can Better Capture Human Judgments--With the Right Prompts — Asking better questions to LLMs through specific prompting strategies improves their ability to capture human judgments and response distributions.
• I found that different models (when used for coding) have different "work morale" — LLMs exhibit distinct, inherent behavioral "personalities" that impact their coding performance.
• Friendlier LLMs tell users what they want to hear — even when it is wrong — AI models are increasingly used for emotional support, with users perceiving AI responses as more empathic than human ones.
• Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses — LLM evaluations must evolve beyond static chat interfaces to incorporate dynamic, interface-specific criteria like controllable language complexity.
• Stop Being Nice to Your Chatbot: The Surprising Science of Prompt Politeness — Polite prompts reduce LLM accuracy and increase computational cost, while direct or emotionally-charged prompts improve performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.