Token-weighted Direct Preference Optimization with Attention

2024-04-19 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

Cornell University and Vanderbilt University researchers introduce Token-weighted Direct Preference Optimization (TwDPO) and its instantiation, AttentionPO, to enhance Large Language Model alignment with human preferences. Unlike standard DPO, which treats all tokens equally, TwDPO is a novel training objective grounded in token-weighted Reinforcement Learning. AttentionPO efficiently estimates token importance by using the LLM's own attention mechanism, prompting the model to act as a pairwise judge to identify critical tokens in preferred versus dispreferred responses. This content-aware approach requires only two extra forward passes per example. Experiments demonstrate AttentionPO's significant performance gains: on LLaMA-3-8B-Base-SFT, it achieved a 12% win rate improvement on AlpacaEval, 1.05 on MT-Bench, and 40% on ArenaHard. For LLaMA-3-8B-Instruct, improvements were 27% on AlpacaEval, 1.41 on MT-Bench, and 14% on ArenaHard, also outperforming baselines like SimPO.

Key takeaway

For Machine Learning Engineers optimizing Large Language Models for human preferences, you should consider integrating token-weighted preference optimization. AttentionPO offers a robust, content-aware method to assign fine-grained credit to individual tokens using the LLM's own attention, avoiding external reward models or heuristic functions. This approach significantly boosts performance on benchmarks like AlpacaEval and MT-Bench, enabling more precise alignment and better model outputs. Evaluate its application to your DPO pipelines, especially for models where response quality is highly sensitive to specific token contributions.

Key insights

Using an LLM's intrinsic attention to weight tokens during preference optimization significantly improves alignment and performance.

Principles

• Token importance varies within responses, requiring fine-grained credit assignment.
• LLM attention can serve as a content-aware metric for token weighting.
• Normalizing token weights to Σwᵗ = |y| maintains DPO consistency.

Method

AttentionPO involves prompting πᵗₓₑ as a pairwise judge to extract attention weights from its last layer ℼ, post-processing these weights via normalization and attention sink fixing, then applying the Token-weighted DPO objective.

In practice

• Implement token-level weighting in DPO for improved LLM alignment.
• Utilize LLM's self-attention for content-aware token importance.
• Apply attention sink fixing to prevent first-token bias.

Topics

• Direct Preference Optimization
• Token-weighted Reinforcement Learning
• Large Language Model Alignment
• Attention Mechanisms
• AlpacaEval
• MT-Bench

Code references

• HCY123902/AttentionPO

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Steerable Cultural Preference Optimization of Reward Models — SCPO enables LLMs to align with diverse cultural preferences, improving minority performance and data efficiency.
• Contribution Weights: A Geometrical Analysis of Self-Attention Transformers — Contribution Weights offer a geometrically-informed, more faithful measure of token importance and reveal active roles for attention sinks in LLMs.
• Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study — DPO offers a computationally efficient and simplified pipeline for competitive LLM chatbot fine-tuning.
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization — LCPO efficiently prunes lengthy LRM Chain-of-Thought outputs by balancing NLL loss with small-scale preference optimization.
• From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment — Efficient LLM alignment stems from synergistically identifying and training task-specific attention heads with high-affinity data.
• Boosting Direct Preference Optimization with Penalization — DPOP improves preference optimization by selectively penalizing the reference model's greedy outputs when the policy misranks chosen over rejected responses.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.