SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Intermediate, quick

Summary

The SPLIT benchmark, comprising 500 prompts, evaluates Large Language Model consistency in generating emotionally grounded responses across English and Ukrainian, specifically for crisis-related situations. It assesses three technically diverse LLMs—Gemini-2.5-Flash, LLaMA-3.3-70B-Instruct, and DeepSeek-V3—across five categories: Stress, Panic, Loneliness, Internal Displacement, and Tension. Evaluation focuses on Empathetic Accuracy, Linguistic Naturalness, and Contextual & Cultural Grounding. Findings indicate that Gemini-2.5-Flash and LLaMA-3.3-70B-Instruct show degradation when transitioning to Ukrainian, while DeepSeek-V3 maintains comparative stability. Furthermore, human and AI evaluators exhibit weak agreement on empathy and naturalness but diverge significantly on cultural grounding, highlighting that generating Ukrainian text does not equate to providing culturally appropriate emotional support.

Key takeaway

For NLP Engineers or AI Ethicists deploying Large Language Models in cross-lingual emotional support or crisis contexts, you must prioritize rigorous human-centered evaluation for cultural grounding. Your current multilingual LLMs, like Gemini-2.5-Flash and LLaMA-3.3-70B-Instruct, likely degrade in low-resource languages such as Ukrainian, even if they produce fluent text. Focus on culturally tailored benchmarks and avoid over-reliance on AI evaluators for nuanced emotional and cultural appropriateness.

Key insights

Large Language Models often degrade in cross-lingual emotional support, particularly cultural grounding, when transitioning to low-resource languages like Ukrainian.

Principles

• Cross-lingual emotional support performance in LLMs can degrade significantly in low-resource languages.
• Cultural grounding in LLM responses is a distinct and challenging dimension, often missed by AI evaluators.
• Generating text in a target language does not guarantee culturally appropriate emotional support.

Method

The SPLIT benchmark evaluates LLM consistency in emotionally grounded responses using 500 prompts across five crisis categories and three dimensions: Empathetic Accuracy, Linguistic Naturalness, and Contextual & Cultural Grounding, also exploring LLM-as-a-jury reliability.

In practice

• Design benchmarks with stronger cultural tailoring for multilingual LLMs.
• Prioritize human-centered evaluation for LLMs deployed in sensitive emotional support contexts.

Topics

• Cross-Lingual LLMs
• Emotional Support AI
• Cultural Grounding
• LLM Evaluation
• SPLIT Benchmark
• Ukrainian Language

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual, multicultural, and grounded reasoning on real-world tasks, revealing significant performance gaps.
• Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark — Monolingual LLMs for under-resourced languages require specific evaluation against multilingual counterparts.
• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training — Multilingual post-training significantly improves LLM performance and cross-lingual generalization, even with minimal non-English data.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — MADE offers a structured, agentic approach for fine-grained, multilingual LLM evaluation diagnosis, moving beyond simple benchmark scores.
• A Multilingual Paradigm for AI Created by All, for All — Multilingual AI evaluations must be culturally nuanced, context-specific, and independently verified to ensure equitable global AI performance.
• Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research — LLMs offer transformative potential for low-resource language research in humanities, despite significant data and ethical challenges.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.