Beyond Accuracy: A Structured Error Analysis of Multilingual LLMs on Marathi Script Variation and Syntax

2026-06-26 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Tejas Patil and Barnali Chetia conducted a structured error analysis of four multilingual large language models—Llama-3.1-8B, Llama-3.3-70B, Mistral-7B, and Qwen3-32B—on Marathi, a language spoken by over 83 million people. Their study addressed the lack of systematic evaluation beyond surface-level benchmarks for Marathi, which possesses significant morphological complexity. Using a manually curated dataset, the models were tested across Devanagari versus Romanized script, Marathi-English code-mixing, and specific syntactic structures like SOV word order and vibhakti case markers. Key findings include a performance drop of 7.9% to 20.5% on Romanized input and consistent replacement of Marathi vibhakti case markers with Hindi equivalents. The negative subjunctive marker "nasta" was also ignored by all models. These results expose structural deficiencies in how current multilingual LLMs process morphologically rich, low-resource Indic languages.

Key takeaway

For NLP Engineers developing multilingual LLMs for Indic languages, this analysis highlights critical performance gaps. You should prioritize dedicated Marathi pretraining data to address specific morphological challenges like vibhakti case markers and the "nasta" subjunctive. Ensure your models develop distinct internal representations for languages like Hindi and Marathi, rather than conflating them. Additionally, consider the significant performance drop on Romanized input, suggesting a focus on Devanagari script processing for optimal results.

Key insights

Multilingual LLMs exhibit structural gaps in handling Marathi's morphological complexity, particularly script variations and distinct grammatical markers.

Principles

• Multilingual training can conflate distinct languages.
• Romanized input degrades Indic language LLM performance.
• Morphological complexity requires dedicated pretraining data.

Method

Evaluate LLMs using a manually curated dataset across script variations, code-mixing, and syntactic structures (SOV, vibhakti, verb agreement, postpositions) via translation, similarity, grammaticality, and case marker tasks.

In practice

• Prioritize Devanagari script for Marathi LLM inputs.
• Develop dedicated pretraining for Marathi morphology.
• Ensure distinct internal representations for Indic languages.

Topics

• Multilingual LLMs
• Marathi Language Processing
• Error Analysis
• Indic Languages
• Morphological Analysis
• Script Variation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages — IndicDB reveals a significant "Indic Gap" in LLM Text-to-SQL performance for Indian languages compared to English.
• Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions — LLMs consistently underperform on Romanized Code-Mixed instructions, with performance degrading as code-mixing density increases.
• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.
• UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding — Native, culturally-grounded benchmarks reveal significant LLM knowledge gaps in non-English, humanities-focused content.
• Building Singlish2Sinhala: A Machine Learning Approach to Sinhala Transliteration — Informal transliteration systems like Singlish present unique NLP challenges due to spelling inconsistencies.
• Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation — Multilingual-IRT extends Item Response Theory to efficiently evaluate LLMs across languages, identifying translation errors and cultural bias.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.