MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

2026-04-20 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

MORPHOGEN is a new large-scale benchmark dataset designed to evaluate the gender-aware morphological generation capabilities of multilingual large language models (LLMs). It focuses on three typologically diverse grammatically gendered languages: French, Arabic, and Hindi. The primary task, GENFORM, challenges models to rewrite a first-person sentence into the opposite gender while maintaining its original meaning and structure. Researchers constructed a high-quality synthetic dataset for these languages and used it to benchmark 15 popular multilingual LLMs, ranging from 2B to 70B parameters. The initial findings indicate substantial deficiencies in how current models manage morphological gender, highlighting a critical area for improvement in inclusive and morphology-sensitive natural language processing.

Key takeaway

For research scientists developing or deploying multilingual LLMs, understanding gender-aware morphological generation is crucial. Your models likely have significant gaps in handling grammatical gender in languages like French, Arabic, and Hindi, which can lead to biased or incorrect outputs. You should integrate benchmarks like MORPHOGEN into your evaluation pipelines to identify and address these limitations, ensuring more inclusive and accurate language model performance.

Key insights

MORPHOGEN evaluates LLM gender-aware morphological generation in French, Arabic, and Hindi via a sentence rewriting task.

Principles

• Grammatical gender impacts verb conjugation and pronouns.
• LLMs show significant gaps in handling morphological gender.

Method

The GENFORM task requires models to rewrite first-person sentences into the opposite gender, preserving meaning and structure, across French, Arabic, and Hindi using a synthetic dataset.

In practice

• Benchmark LLMs on gender-aware generation.
• Diagnose model limitations in morphological agreement.

Topics

• MORPHOGEN Benchmark
• Gender-aware Generation
• Morphological Agreement
• Multilingual LLMs
• Grammatical Gender

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation — MORPHOGEN evaluates LLM gender-aware morphological generation across French, Arabic, and Hindi, revealing performance gaps and biases.
• ProText: A Benchmark Dataset for Measuring (Mis)gendering in Long-Form Texts — ProText measures gendering and misgendering in LLM text transformations, revealing systematic gender bias beyond binary assumptions.
• A Scalable Tool for Measuring Manner and Result Verbs in Developmental Language Research — A RoBERTa classifier, trained on LLM-generated annotations, accurately distinguishes manner and result verbs at scale.
• Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew — Tokenization strategy critically impacts Transformer models' ability to represent complex morphology, especially in non-concatenative languages.
• Gender Identification in Brazilian Portuguese Product Reviews: A Comparative Study of Classical Models, BERT, and LLMs — Gender identification in Brazilian Portuguese Amazon reviews shows BERT outperforming LLMs, with performance varying by author gender and product domain.
• GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German — The GRUFF dataset reveals German LLMs struggle with neopronouns and distractor robustness, despite strong grammatical gender agreement.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.