Taken into Consideration: An Evaluation of Estimation Biases by Race, Gender, and Region in Large Language Models in Brazilian Portuguese

2026-04-12 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study evaluated social biases in Brazilian Portuguese within large language models, specifically GPT-4o, GPT-4o-mini, Sabiá-3, and Sabiázinho-3. Researchers used an "esteem" metric to quantify the models' respect and deference towards various demographic groups, including those with explicit markers for gender, race, and Brazilian region. The evaluation was conducted both with and without a jailbreaking technique to circumvent moderation restrictions. Findings indicate that these models consistently exhibit systematic patterns of differentiated valuation, reproducing esteem biases linked to gender, race, and regional markers. Subjects with emphasized social markers, particularly racial ones, generally received lower esteem. The jailbreaking technique produced inconsistent results, sometimes amplifying and other times reducing these esteem differences.

Key takeaway

For research scientists and engineers developing or deploying LLMs for Portuguese-speaking populations, you must rigorously evaluate models like GPT-4o and Sabiá-3 for inherent esteem biases related to race, gender, and region. Your bias mitigation strategies should account for the inconsistent effects of jailbreaking techniques, as they may exacerbate or alleviate existing biases, requiring careful, context-specific testing.

Key insights

Large language models exhibit systematic esteem biases related to gender, race, and region in Brazilian Portuguese.

Principles

• LLMs reproduce social biases.
• Racial markers correlate with lower esteem.
• Jailbreaking impact is inconsistent.

Method

The study identified social biases in LLMs using an "esteem" metric, evaluating models' deference to demographic groups with and without moderation circumvention (jailbreaking) in Brazilian Portuguese.

In practice

• Evaluate LLMs for esteem biases.
• Test jailbreaking impact on bias.
• Focus on racial bias mitigation.

Topics

• LLM Bias Evaluation
• Esteem Metric
• Brazilian Portuguese NLP
• Gender and Race Bias
• Regional Bias

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Bias and Justice in Language Models: Evidence from a Linguistically, Socially, and Culturally Asymmetric Literature — LLM bias research is skewed towards English and binary gender, lacking intersectional and multilingual real-world analyses.
• IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages — Culturally-grounded benchmarks are crucial for evaluating LLM bias in diverse, multilingual contexts like Indonesia.
• Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs — Multilingual LLMs exhibit implicit US-centric and population-driven biases, exacerbated by instruction tuning.
• A Systematic Analysis of Biases in Large Language Models — LLMs, despite alignment efforts, inherit and perpetuate diverse biases from human-generated training data.
• Extending the Contact Hypothesis: Cross-Linguistic Evaluation of Religion and Nationality Bias When Prompting LLMs in German and Icelandic — LLMs exhibit religion and nationality biases, requiring cross-linguistic evaluation for fairness.
• SPAGBias: Uncovering and Tracing Structured Spatial Gender Bias in Large Language Models — LLMs encode structured spatial gender biases that exceed real-world distributions, impacting urban planning applications.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.