Does Bigger Mean Funnier? Evaluating Humor Generation Across the Qwen3 Model Family

2026-06-27 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A study evaluated humor generation across five Qwen3 model variants, ranging from 8B to 235B parameters (dense and MoE), by generating jokes across 50 themes. The research investigated whether scaling model parameters improves humor quality and compared LLM versus human evaluations on this subjective creative task. An automated judge consistently ranked larger models as funnier, showing a perfect monotonic relationship between parameter count and win rate. However, human annotators found no significant aggregate difference in humor quality among the models. When analysis was restricted to themes where human annotators agreed, a significant preference for the largest model was observed (p = 0.039), indicating that scaling effects might exist but are obscured by a "quality floor." Crucially, the automated judge exhibited severe positional and length biases compared to human evaluators, suggesting LLMs can systematically distort quality differences in subjective assessments.

Key takeaway

For NLP Engineers evaluating creative LLM outputs like humor, you should prioritize human judgment over automated metrics. Automated judges, especially LLM-based ones, exhibit severe positional and length biases that can distort perceived quality differences. If you rely solely on automated evaluations, you risk misinterpreting model performance and overlooking subtle scaling effects. Implement robust human annotation protocols to accurately assess subjective tasks and ensure your models genuinely improve.

Key insights

LLM-based humor evaluation is unreliable due to inherent biases, masking true scaling effects.

Principles

• Human evaluation is crucial for subjective tasks.
• Automated judges introduce significant biases.
• Scaling effects can be masked by a "quality floor."

Method

The study used an ablation design with five Qwen3 variants (8B–235B) to generate jokes across 50 themes, evaluated by both automated judges and human annotators.

In practice

• Prioritize human evaluation for creative LLM outputs.
• Scrutinize automated evaluation for positional/length biases.
• Consider "quality floor" when interpreting scaling results.

Topics

• Humor Generation
• LLM Evaluation
• Qwen3 Model Family
• Model Scaling
• Human-in-the-Loop
• Evaluation Bias

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Human Evaluation of Large Language Models: How Good is Hugging Face’s BLOOM? — Human evaluation is crucial for assessing large language models' real-world performance beyond flawed academic benchmarks.
• Investigating Counterfactual Unfairness in LLMs towards Identities through Humor — LLMs exhibit counterfactual unfairness in humor, showing bias based on speaker/addressee identity swaps.
• Re-defining Humor Data Objects for AI Humor Research — AI humor research improves by defining humor as a social interaction with context and explanations, not just a binary state.
• I built a public voting benchmark where models have to make memes out of daily news — Memebench evaluates LLM humor and creativity by generating memes from daily news for public A/B voting.
• # I Got Tired of Saying “This Response Is Better” Without Being Able to Prove It — Combining human domain expertise with automated pattern detection improves LLM evaluation accuracy and provides objective vocabulary for subjective judgments.
• Sonnet 5 review: I ran 64 generations to find out if it's worth it — Benchmarking frontier LLMs requires combining automated metrics with human "taste" to capture nuanced performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.