Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

2026-05-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A new approach addresses the "alignment tax" in extending large language models (LLMs) to low-resource languages, where improving one language often degrades general capabilities. This trade-off is attributed to the rigidity of supervised fine-tuning (SFT), which relies on token-level imitation from limited, biased data. Researchers propose a semantic-space alignment paradigm utilizing Group Relative Policy Optimization (GRPO), optimizing models with embedding-level semantic rewards instead of likelihood maximization. This method promotes meaning preservation through flexible realizations, leading to controlled updates that minimize interference with pretrained knowledge. Evaluated on Tibetan-Chinese machine translation and Tibetan headline generation, the approach significantly mitigates alignment tax, preserving general competence more effectively than SFT, and yields higher semantic quality and user preference in open-ended generation.

Key takeaway

For research scientists developing LLMs for low-resource languages, you should consider adopting reinforcement learning with semantic rewards. This approach offers a safer and more reliable pathway for language expansion by mitigating the "alignment tax" and preserving general competence, unlike traditional supervised fine-tuning. Your models will likely achieve higher semantic quality and more transferable representations.

Key insights

Semantic-space alignment with reinforcement learning mitigates "alignment tax" in low-resource language expansion for LLMs.

Principles

• Rigid SFT causes "alignment tax."
• Semantic rewards preserve meaning flexibly.

Method

Optimize LLMs using embedding-level semantic rewards via Group Relative Policy Optimization (GRPO) to encourage meaning preservation and reduce destructive interference with pretrained knowledge.

In practice

• Apply GRPO for low-resource language expansion.
• Use semantic rewards for flexible model updates.

Topics

• Reinforcement Learning
• Semantic Rewards
• Low-Resource Languages
• Large Language Models
• Alignment Tax

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax — Semantic-space alignment with reinforcement learning mitigates "alignment tax" in low-resource language expansion.
• One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models — Reward Models (RMs), crucial for aligning language models (LMs) with human preferences, are shown to still suffer from persistent biases like length, sycophancy, and overconfidence, even in state-of-the-art models.
• Reinforcement Learning without Ground-Truth Solutions can Improve LLMs — RiVER trains LLMs on score-based tasks without ground-truth, using calibrated reward shaping for general coding ability.
• Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning — CoUR uses LLMs and uncertainty quantification to streamline RL reward function design and evaluation.
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments — Automated environment generation and verifiable rewards significantly boost LLM tool-use performance via reinforcement learning.
• S-SPPO: Semantic-Calibrated Self-Play Preference Optimization — S-SPPO stabilizes LLM preference alignment by semantically calibrating self-play optimization to prevent policy degeneration from indistinguishable responses.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.