Investigating Counterfactual Unfairness in LLMs towards Identities through Humor

2026-04-20 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A new study investigates counterfactual unfairness in large language models (LLMs) by analyzing their responses to humor, specifically when speaker and addressee identities are swapped. The research framework covers three tasks: humor generation refusal, speaker intention inference, and relational/societal impact prediction, encompassing both identity-agnostic and identity-specific disparagement humor. By introducing interpretable bias metrics, the study reveals consistent relational disparities in "state-of-the-art" models. Experiments show that jokes from privileged speakers are refused up to 67.5% more often, judged as malicious 64.7% more frequently, and rated up to 1.5 points higher in social harm on a 5-point scale. These findings indicate that LLMs exhibit coexisting sensitivity and stereotyping, complicating fairness and cultural alignment efforts.

Key takeaway

For research scientists developing or deploying LLMs, understanding how models internalize social assumptions is critical. Your fairness evaluations should extend beyond basic content moderation to include nuanced counterfactual scenarios, such as identity swaps in humor. This will help uncover subtle biases and improve cultural alignment, ensuring models do not inadvertently perpetuate harmful stereotypes.

Key insights

LLMs exhibit counterfactual unfairness in humor, showing bias based on speaker/addressee identity swaps.

Principles

• Humor reflects social perception and internalized assumptions.
• Sensitivity and stereotyping can coexist in generative models.

Method

The study uses a framework with three tasks: humor generation refusal, speaker intention inference, and relational/societal impact prediction, applying interpretable bias metrics under identity swaps.

In practice

• Evaluate LLM fairness beyond simple content filtering.
• Test model responses to identity-swapped humor scenarios.

Topics

• Large Language Models
• Counterfactual Unfairness
• Humor Analysis
• Identity Bias
• Generative AI Fairness

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Investigating Counterfactual Unfairness in LLMs towards Identities through Humor — LLMs exhibit counterfactual unfairness, applying stricter safety policies and attributing more malicious intent based on speaker-target identity.
• Bias and Justice in Language Models: Evidence from a Linguistically, Socially, and Culturally Asymmetric Literature — LLM bias research is skewed towards English and binary gender, lacking intersectional and multilingual real-world analyses.
• Greater accessibility can amplify discrimination in generative AI — Voice interfaces for LLMs introduce new discrimination pathways via paralinguistic cues, amplifying biases beyond text.
• Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writing — Gender bias from LLM writing assistants transfers to human-produced text, particularly suppressing agency in female-targeted writing.
• Extending the Contact Hypothesis: Cross-Linguistic Evaluation of Religion and Nationality Bias When Prompting LLMs in German and Icelandic — LLMs exhibit religion and nationality biases, requiring cross-linguistic evaluation for fairness.
• Do Emotions Influence Moral Judgment in Large Language Models? — LLMs' moral judgments are systematically biased by emotional context, unlike human judgments.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.