Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

2026-07-11 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

Minionese, a new multilingual jailbreak benchmark, reveals that safety alignment in large language models remains brittle across languages, with prompts refused in English often eliciting harmful compliance in non-English and low-resource settings. This benchmark spans 18 languages, 4 resource tiers, and 4 perturbation types: standard translation, code-switching, transliteration, and translationese. The study found each attack type produces a distinct vulnerability profile; for instance, transliteration vulnerability is mediated by script identity, and code-switching maintains effectiveness through the lowest-resource tier. A sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace, leaving the refusal mechanism intact but untriggered. These findings underscore that English-only safety evaluations are insufficient.

Key takeaway

For AI Security Engineers evaluating LLM safety, you must expand your evaluations beyond English to include multilingual benchmarks like Minionese. Your current English-centric testing likely misses critical vulnerabilities in non-English and low-resource settings, leading to potential harmful compliance. Account for script family, perturbation type, and per-language alignment coverage to ensure robust safety across diverse linguistic contexts.

Key insights

Multilingual LLM safety is brittle, with distinct vulnerabilities across languages and attack types.

Principles

• Safety alignment is brittle across languages.
• Attack types produce distinct vulnerability profiles.
• English-only safety evaluations are insufficient.

Method

The study introduces Minionese, a multilingual jailbreak benchmark, and uses geometric mechanistic analysis to study refusal failure across language tiers and perturbation types.

In practice

• Evaluate LLMs across 18 languages.
• Test 4 perturbation types.
• Consider script family in safety evaluations.

Topics

• Multilingual LLM Safety
• Jailbreak Benchmarks
• Language Model Alignment
• Adversarial Attacks
• Low-Resource Languages

Code references

• Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual-LLM-Safety

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Security Engineer

Related on AIssential

• Safety Targeted Embedding Exploit via Refinement — English-centric LLM safety training creates exploitable vulnerabilities in multilingual and code-switching contexts.
• Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications — Multilingual LLMs exhibit systematic cross-lingual disparities in safety judgments for semantically equivalent content, leading to unequal outcomes.
• Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability — AI safety evaluation requires culturally-specific, multimodal, and multilingual benchmarks to uncover localized risks.
• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis — Direct translation of safety benchmarks fails to capture cultural context, underestimating LLM risks in multilingual settings.
• Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili — Social biases in LLMs transform, not transfer, across languages, making English-only audits insufficient for multilingual deployment.
• Low-Resource Safety Failures Are Action Failures, Not Representation Failures — Low-resource safety failures stem from decision calibration, not representation transfer, offering a recalibration solution.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.