CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

CCBench is a new framework designed to evaluate cultural competence in large language models (LLMs) by treating culture as a continuum of norm adherence rather than a binary state. Its instantiation, CCBench-Health, uses 60 theoretically grounded personas across six cultures, each engaging in 18 realistic dialogues, evaluated on 52 authentic healthcare questions, totaling 3,120 unique interactions. Benchmarking five leading models revealed that even the best achieve culturally appropriate responses only 20-30% of the time. Explicitly prompting with cultural Chain-of-Thought (CoT) modestly improved performance by 3-5%. Models performed better when personas avoided cultural norms, indicating a "Western default" bias, particularly evident in the Afghan context (Avg: 8.8%). Models sometimes adapt more readily to implicit conversational styles than explicit cultural practices.

Key takeaway

For machine learning engineers developing global-facing LLMs, recognize that current models default to Western norms and struggle with nuanced cultural adaptation. Prioritize training data diversity and develop techniques that enable active accommodation of non-Western cultural values, rather than just stereotype avoidance. Consider integrating explicit cultural reasoning scaffolds and robust evaluation against benchmarks like CCBench-Health to ensure equitable and trustworthy AI for diverse user populations.

Key insights

LLMs exhibit a "Western default" bias, struggling to adapt to implicitly signaled cultural norms, especially non-Western ones.

Principles

• Culture is a continuum of norm adherence, not binary.
• Implicit cultural cues are critical for true competence.
• LLMs show asymmetry: better at avoiding than following norms.

Method

CCBench evaluates LLMs by generating personas with varied norm adherence, creating conversational histories with implicit cues, and using a checklist-based evaluation for culturally appropriate responses to queries.

In practice

• Explicit cultural CoT improves LLM performance modestly (3-5%).
• Models struggle with deeply embedded cultural users.
• Performance varies significantly across different cultures.

Topics

• Cultural Competence
• Large Language Models
• Healthcare AI
• Bias Evaluation
• Norm Adherence
• Persona Simulation

Code references

• LasseRegin/medical-question-answer-data

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs — Multilingual LLMs exhibit implicit US-centric and population-driven biases, exacerbated by instruction tuning.
• Korean Culture into LLM Alignment: Toward Cultural Coherence — LLM cultural alignment requires defining culturally coherent responses, not just suppressing harmful ones.
• CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine — LLMs exhibit a "humility deficit" in medical contexts, struggling with ambiguity and preferring incorrect answers over admitting uncertainty.
• Mod-Guide: An LLM-based Content Moderation Feedback System to Address Insensitive Speech toward Indigenous Ethnic and Religious Minority Communities — Integrating culturally grounded narratives via RAG enhances LLM sensitivity to minority perspectives in content moderation.
• Extending the Contact Hypothesis: Cross-Linguistic Evaluation of Religion and Nationality Bias When Prompting LLMs in German and Icelandic — LLMs exhibit religion and nationality biases, requiring cross-linguistic evaluation for fairness.
• Toward Culturally Aligned LLMs through Ontology-Guided Multi-Agent Reasoning — Ontology-guided multi-agent reasoning with demographic grounding improves LLM cultural alignment and interpretability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.