Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

2026-07-16 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A study investigated language bias in Large Language Models (LLMs) for code generation, specifically how different natural languages in prompts affect code quality. Researchers analyzed GPT-4o mini, DeepSeek, and Claude across 460 coding tasks (230 Python, 230 Java). They manually translated English prompts into Chinese, Hindi, Spanish, and Italian, preserving technical meaning. Evaluation covered functional correctness via test pass rates, structural quality using code metrics, static analysis issues, and lexical characteristics like identifier and comment languages. Key findings indicate that English prompts do not consistently produce the best functional correctness or code quality, the impact of prompt language varies by programming language and LLM, and generated code often mixes English with the prompt language in comments and string literals.

Key takeaway

For Machine Learning Engineers developing multilingual code generation systems, you should not assume English prompts will consistently deliver optimal code quality or functional correctness. Your LLM's performance is sensitive to the prompt's natural language, varying by programming language and model. Test your systems with diverse prompt languages relevant to your user base to ensure robust and high-quality code output across all supported languages.

Key insights

Prompt language significantly influences LLM code generation quality, with English not always yielding superior results.

Principles

• Prompt language impact depends on both the programming language and the LLM.
• Generated code frequently mixes English with the prompt language.

Method

Manually curate and translate 460 English coding prompts into four other languages, then evaluate LLM-generated Python and Java code across multiple quality dimensions.

In practice

• Curated multilingual benchmark for language bias studies.
• Insights for robust multilingual code generation systems.

Topics

• Large Language Models
• Code Generation
• Language Bias
• Multilingual Prompts
• Code Quality
• GPT-4o mini

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality — LLM code generation quality is biased by prompt language, with English not always superior.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs — Prompt syntax, specifically constituent type and position, fundamentally shapes the security of LLM-generated code.
• DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation — Dialectal robustness and generation in LLMs are dissociated, requiring richer rewards and resources for authentic dialectal output.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs demonstrate effective multilingual reasoning across 43 languages, with machine translations proving as effective as human ones.
• Code Is More Than Text: Uncertainty Estimation for Code Generation — Code generation uncertainty estimation requires specialized methods accounting for code's unique properties, outperforming natural language-derived baselines.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.