SemEval-2026 Task 1: MWAHAHA, Models Write Automatic Humor And Humans Annotate

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

SemEval-2026 Task 1, named MWAHAHA (Models Write Automatic Humor And Humans Annotate), introduced the first shared task focused on general-purpose humor generation. This task required systems to create short jokes in English, Spanish, and Chinese under lexical or topical constraints (Subtask A), and to generate humorous captions for GIFs (Subtask B). To ensure fairness and prevent memorization, jokes had to meet specific criteria, such as using infrequent word pairs or relating to recent news. Evaluation involved pairwise human preference judgments in a Chatbot Arena-style setup, resulting in Elo-based rankings. The task attracted 309 registered users, with 37 teams submitting systems employing diverse NLP techniques, including generate-then-rank pipelines and reinforcement learning. Notably, a Gemini 2.5 Flash baseline, using simple prompts, tied for first place across all subtasks, often outperforming or matching more complex multi-stage pipelines. The organizers released all evaluation data, prompts, and leaderboard results.

Key takeaway

For NLP engineers developing humor generation systems, you should re-evaluate the complexity of your pipelines. The strong performance of a simple Gemini 2.5 Flash baseline suggests that sophisticated multi-stage approaches may offer diminishing returns. Prioritize optimizing prompt engineering for powerful large language models before investing heavily in intricate architectures. Your efforts might be better spent refining prompts and leveraging existing LLM capabilities, rather than building complex custom solutions that only marginally surpass simple baselines.

Key insights

Simple prompting with advanced LLMs often matches or exceeds complex humor generation pipelines.

Principles

• Human preference judgments are effective for humor evaluation.
• Baseline LLM performance can challenge elaborate NLP pipelines.
• Humor generation benefits from diverse linguistic and contextual constraints.

Method

Systems generate jokes under lexical/topical constraints or humorous GIF captions, then human evaluators provide pairwise preferences for Elo-based ranking.

In practice

• Utilize Gemini 2.5 Flash with simple prompts for humor tasks.
• Explore human preference judgments for model evaluation.
• Access released evaluation data and prompts for research.

Topics

• Computational Humor
• Humor Generation
• SemEval
• Large Language Models
• Gemini 2.5 Flash
• Human Evaluation
• Prompt Engineering

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Re-defining Humor Data Objects for AI Humor Research — AI humor research improves by defining humor as a social interaction with context and explanations, not just a binary state.
• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.
• Humor Style Drives Laughter, Topic Shapes Acceptability: Evaluating Bilingual Personal and Political Robot-Delivered AI Jokes — Aggressive and Affiliative humor styles enhance robot-delivered joke funniness, while person-related topics improve appropriateness.
• Detecting HIV-Related Stigma in Clinical Narratives Using Large Language Models — LLMs can effectively detect HIV-related stigma in clinical notes, with encoder models outperforming generative models in zero-shot settings.
• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.
• #1 on memory benchmark LongMemEval with Gemini Flash, not Pro [R] — Cognitive science-informed retrieval architectures can significantly enhance conversational memory performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.