Token-Level Entropy Reveals Demographic Disparities in Large Language Models

2024-04-09 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

The study "Token-Level Entropy Reveals Demographic Disparities in Large Language Models" investigates homogeneity bias in Large Language Models (LLMs) and Vision-Language Models (VLMs), proposing it stems from systematic differences in token sampling probability distributions during inference. Researchers analyzed GPT-4 Turbo and Llama-3.2, finding that tokens were sampled more deterministically when generating 50-word stories about marginalized groups, specifically Black Americans and women, compared to White Americans and men. This was quantified using three uncertainty measures: entropy, perplexity, and probability of differentiation. While these models showed lower uncertainty for marginalized groups, this pattern did not consistently replicate across all tested VLMs, including GPT-4o mini and Ovis1.6. Some models even exhibited inverse patterns, where marginalized groups had higher sampling uncertainty. The findings suggest differential sampling uncertainty contributes to homogeneity bias in some models, but not universally, indicating diverse underlying mechanisms.

Key takeaway

For Machine Learning Engineers evaluating model fairness, you should investigate token-level sampling distributions directly. This research shows that homogeneity bias in models like GPT-4 Turbo and Llama-3.2 stems from more deterministic token sampling for marginalized groups. Relying solely on output analysis might miss these underlying mechanisms. You must assess uncertainty measures like entropy and perplexity to identify and address differential treatment at the inference stage, recognizing that bias mechanisms vary across models.

Key insights

Homogeneity bias in LLMs can arise from more deterministic token sampling for marginalized groups.

Principles

• Homogeneity bias manifests as differential token sampling uncertainty.
• Bias mechanisms vary significantly across different LLM architectures.
• Token sampling uncertainty decreases as text generation progresses.

Method

Researchers used facial stimuli to prompt VLMs to generate 50-token stories. They collected top 20 log probabilities at each token position and computed entropy, perplexity, and probability of differentiation to measure uncertainty.

In practice

• Evaluate LLM outputs for differential token sampling uncertainty.
• Compare uncertainty measures across demographic groups.
• Investigate model-specific bias mechanisms.

Topics

• Homogeneity Bias
• Large Language Models
• Vision-Language Models
• Token Sampling Uncertainty
• Algorithmic Bias
• GPT-4 Turbo

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, AI Ethicist, Machine Learning Engineer

Related on AIssential

• Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoders — LLMs may infer demographic attributes from subtle linguistic cues, even before explicit output.
• An Information-Theoretic Study of RLHF-Induced Uniformity in Large Language Model Outputs — RLHF impacts information uniformity and consistency in Large Language Model outputs.
• Growing a Tail: Increasing Output Diversity in Large Language Models — LLMs produce narrow outputs; simple interventions can increase diversity but often fall short of human levels.
• Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models — LLMs widely violate statistical self-consistency, often performing better when aggregating fine-grained subpopulation estimates than direct population estimates.
• Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning — A new metric quantifies LLM lexical bias from preference learning without manual curation.
• Emergent retokenization symmetry in large language models: phenomenology and applications — Large language models partially develop retokenization symmetry, allowing diverse outputs from semantically equivalent input segmentations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.