Navigating the Joke Space: Towards Automated Originality Assessment of AI-Generated Humor

2026-06-27 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A study validates automated, corpus-based methods for quantifying joke originality, focusing on "topic handles" derived from the General Theory of Verbal Humor. Researchers used a reference corpus of one million English jokes from Reddit to compute three variants of Pointwise Mutual Information (PMI) and two embedding-based measures: handle-level conceptual distance and full-text corpus novelty via Sentence-BERT. These methods were evaluated on 400 LLM-generated jokes (200 from GPT-4o and 200 from GPT-5.4) and 80 jokes from the JEST benchmark, all rated by three professional comedians. Corpus novelty and concept distance between the most semantically distant handle pair showed a significant correlation with human originality ratings (ρ = .37). PMI-based measures also exhibited significant, though weaker, associations (ρ = .23–.25) on the most original handle pair. A Lasso-based composite of the top three predictors achieved a cross-validated ρ = .40, accounting for 82% of the theoretically predictable variance. These findings demonstrate the utility of handle-based PMI and semantic novelty metrics as quantitative tools for objectively assessing originality in AI-generated humor.

Key takeaway

For NLP Engineers developing humor generation models, you should integrate handle-based PMI and semantic novelty metrics into your evaluation pipelines. These quantitative tools, particularly corpus novelty and conceptual distance, offer objective measures that correlate with human originality ratings (ρ = .37). This allows you to systematically assess and improve the creative output of models like GPT-4o and GPT-5.4, moving beyond subjective human feedback alone.

Key insights

Automated metrics using "topic handles" and semantic novelty can quantify AI-generated joke originality, correlating significantly with human assessments.

Principles

• "Topic handles" define joke structure.
• Corpus-based metrics quantify originality.
• Semantic novelty correlates with human humor.

Method

Extract "topic handles" from jokes. Compute PMI (3 variants) and embedding-based measures (conceptual distance, corpus novelty) against a 1M joke corpus. Correlate these metrics with human originality ratings.

In practice

• Use handle-based PMI for originality.
• Apply semantic novelty to AI humor.
• Benchmark LLM jokes with human raters.

Topics

• AI-Generated Humor
• Originality Assessment
• Computational Creativity
• Topic Handles
• Semantic Novelty
• LLM Evaluation

Best for: AI Scientist, Research Scientist, NLP Engineer

Related on AIssential

• Humor Style Drives Laughter, Topic Shapes Acceptability: Evaluating Bilingual Personal and Political Robot-Delivered AI Jokes — Aggressive and Affiliative humor styles enhance robot-delivered joke funniness, while person-related topics improve appropriateness.
• GENIE: A Fine-Grained Measure for Novelty — GENIE offers a fine-grained metric to measure task-specific novelty in LLM outputs, surpassing holistic evaluations.
• I built a public voting benchmark where models have to make memes out of daily news — Memebench evaluates LLM humor and creativity by generating memes from daily news for public A/B voting.
• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• Re-defining Humor Data Objects for AI Humor Research — AI humor research improves by defining humor as a social interaction with context and explanations, not just a binary state.
• Off for adventures — AI image generation still struggles with realistic depiction, often producing humorous artifacts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.