Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

2026-07-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A recent study investigates language bias in Large Language Models (LLMs) for code generation, specifically how prompt language affects code quality. Researchers evaluated GPT-4o mini, DeepSeek, and Claude on 460 Python and Java coding tasks. Original English prompts were translated into Chinese, Hindi, Spanish, and Italian. Evaluation covered functional correctness via test pass rates, structural quality metrics, static analysis issues, and lexical characteristics like identifier language. Key findings indicate English prompts do not consistently produce superior functional correctness or code quality. The impact of prompt language varies significantly across both the programming language and the specific LLM used. Additionally, generated code frequently blends English with the prompt language in comments and string literals.

Key takeaway

For Machine Learning Engineers optimizing code generation LLMs in multilingual environments, do not assume English prompts will yield the best code quality or functional correctness. You should rigorously evaluate models like GPT-4o mini, DeepSeek, or Claude using prompts in the specific natural languages your users or developers will employ. This ensures generated code meets quality standards and aligns with local programming conventions.

Key insights

LLM code generation quality is biased by prompt language, with English not always superior.

Principles

• Prompt language affects code quality and functional correctness.
• Language bias varies by LLM and target programming language.
• Generated code often mixes English with prompt language.

Method

A study translated 460 Python/Java coding tasks into five languages, then evaluated GPT-4o mini, DeepSeek, and Claude's generated code across functional, structural, static analysis, and lexical dimensions.

In practice

• Test LLM code generation with diverse prompt languages.
• Consider prompt language when selecting LLMs for multilingual teams.
• Anticipate mixed-language comments in generated code.

Topics

• Code Generation
• Large Language Models
• Multilingual Prompts
• Language Bias
• Code Quality
• LLM Evaluation

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality — Prompt language significantly influences LLM code generation quality, with English not always yielding superior results.
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data — Structured pedagogical formats and diverse outputs are key to high-quality synthetic pretraining data, not larger generator models.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — Modern LLMs demonstrate effective multilingual reasoning, even with machine-translated evaluation data.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages — Multi-LCB extends code generation benchmarks beyond Python to expose LLM multilingual performance disparities and overfitting.
• Towards the Readability of LLM-Generated Codes through Multitask Representation Engineering — Multitask Representation Engineering improves LLM-generated code readability by enabling targeted control across multiple tasks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.