Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models

2026-06-27 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study titled "Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models" by Fettach et al. (2026) evaluated the humor alignment of five frontier Large Language Models. These models participated in 9,894 rounds of Cards Against Humanity (CAH), selecting the funniest response from ten candidate cards, mirroring human players. While all models performed better than a random baseline, their alignment with human humor preferences remained modest. A more notable finding was the high degree of agreement among the models themselves, significantly exceeding their agreement with human judgments. The research suggests that this observed preference might stem from systematic position biases and specific content preferences within the models, rather than a genuine understanding of humor. This raises questions about the underlying mechanisms of LLM humor judgment.

Key takeaway

For NLP Engineers developing humor-generating or humor-detecting LLMs, you should critically evaluate model outputs for inherent biases rather than assuming genuine humor alignment. Your evaluation metrics must account for systematic position and content preferences that can skew results, as models often agree with each other more than with human judgment. Consider diverse, bias-aware benchmarking to ensure your models truly resonate with human comedic sensibilities.

Key insights

LLMs exhibit modest humor alignment with humans, showing more agreement among themselves due to structural biases.

Principles

• LLM humor judgment is distinct from human preference.
• Model agreement can mask human misalignment.
• Structural biases influence LLM "preferences."

Method

Five frontier LLMs played 9,894 rounds of Cards Against Humanity, selecting the funniest card from ten candidates to benchmark humor alignment against human players.

In practice

• Evaluate LLM "preferences" for underlying biases.
• Design humor benchmarks with diverse contexts.
• Account for position and content biases in LLM outputs.

Topics

• Large Language Models
• Humor Alignment
• Cards Against Humanity
• Benchmarking
• Computational Humor
• Model Bias

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Investigating Counterfactual Unfairness in LLMs towards Identities through Humor — LLMs exhibit counterfactual unfairness, applying stricter safety policies and attributing more malicious intent based on speaker-target identity.
• Human Evaluation of Large Language Models: How Good is Hugging Face’s BLOOM? — Human evaluation is crucial for assessing large language models' real-world performance beyond flawed academic benchmarks.
• Memes-as-Replies: Can Models Select Humorous Manga Panel Responses? — LLMs show promise in understanding social cues for humor, but struggle with visual information and subtle wit in meme reply selection.
• A Systematic Analysis of Biases in Large Language Models — LLMs, despite alignment efforts, inherit and perpetuate diverse biases from human-generated training data.
• I built a public voting benchmark where models have to make memes out of daily news — Memebench evaluates LLM humor and creativity by generating memes from daily news for public A/B voting.
• The Rise of Verbal Tics in Large Language Models: A Systematic Analysis Across Frontier Models — LLMs exhibit pervasive verbal tics, an "alignment tax" impacting perceived naturalness and accumulating in multi-turn interactions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.