DANGNT@SGU at SemEval-2026 Task 1: A Two-Stage Mistral Generator with DistilBERT Reranking for English Humor Generation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, short

Summary

DANGNT@SGU developed a system for SemEval-2026 Task 1, Subtask A, focusing on English text-based humor generation. Their pipeline integrates a two-stage QLoRA-adapted generator, based on mistralai/Mistral-7B-Instruct-v0.2, with a DistilBERT reranker designed to differentiate jokes from non-jokes. The generator undergoes initial adaptation on a raw joke corpus for general humor style, followed by fine-tuning on synthetic instruction–response pairs for "Word Inclusion" and "News Headline" prompts. During inference, the system produces five candidate outputs per input, applies optional lexical constraints for "Word Inclusion" prompts, and then reranks these candidates using the DistilBERT classifier. DANGNT@SGU achieved an Elo score of 962 (95% CI: 926–986), securing 13th place in the official English Subtask A results. The system is practical, reproducible, and built entirely on open models and public data.

Key takeaway

For NLP Engineers developing text generation systems requiring specific stylistic or content constraints, consider adopting a multi-stage generation and reranking architecture. Your team can utilize QLoRA for efficient model adaptation and synthetic data for task-specific tuning, similar to the Mistral-DistilBERT pipeline. This approach enhances output quality and ensures adherence to complex requirements, offering a reproducible framework using open models for diverse generation challenges.

Key insights

A two-stage generator combined with a reranker effectively produces constrained and stylistic humor.

Principles

• Fine-tuning in stages allows for both general style acquisition and task-specific adaptation.
• A discriminative reranker improves output quality by filtering generated text.

Method

A QLoRA-adapted Mistral-7B-Instruct-v0.2 generator is fine-tuned on a joke corpus then synthetic instruction pairs, generating five candidates, which a DistilBERT reranker then scores.

In practice

• Use QLoRA for efficient LLM adaptation on specific tasks.
• Employ a reranker to filter and improve the quality of generated text.
• Fine-tune models on synthetic data for task-specific instruction following.

Topics

• Humor Generation
• Large Language Models
• Mistral-7B
• DistilBERT
• QLoRA
• Text Reranking
• SemEval

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Building an LLM fine-tuning Dataset — Building multi-speaker Reddit datasets for LLM fine-tuning is complex but achievable, with efficient QLoRA methods requiring minimal samples.
• Can Machines Tell Whether Gamers Love or Hate a Game? — Fine-tuning Transformer models like RoBERTa significantly improves sentiment analysis accuracy on imbalanced, complex human language datasets.
• Automatic Generation of Highlights for Academic Paper Via Prompt-based Learning — Prompt-based learning with LLMs can generate academic paper highlights effectively without extensive labeled data.
• The Evolution of LoRA: 15+ Variants You Should Know — LoRA variants offer diverse strategies for efficient, stable, and adaptive AI model fine-tuning.
• LoRi: Low-Rank Distillation for Implicit Reasoning — Hidden-state reasoning trajectories exhibit low-rank structure, enabling efficient distillation for implicit reasoning.
• [P] QLoRA Fine-Tuning of Qwen2.5-1.5B for CEFR English Proficiency Classification (A1–C2) [P] — QLoRA fine-tuning of Qwen2.5-1.5B effectively classifies CEFR English proficiency, despite synthetic data limitations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.