RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization

2024-07-18 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

RLearner-LLM is a novel framework that improves the logical grounding of Large Language Models (LLMs) in knowledge-intensive generation, addressing a key limitation of Direct Preference Optimization (DPO). Standard DPO, using human or LLM judges, suffers from a verbosity bias, favoring fluency over logical correctness, leading to SFT-trained models achieving NLI entailment scores as low as 0.05–0.22. RLearner-LLM introduces Hybrid-DPO, an automated preference construction pipeline fusing a DeBERTa-v3 NLI entailment signal with a verifier LLM score. This eliminates human annotation and overcomes the "alignment tax" that degrades fluency. Evaluated across five academic domains (Biology, Medicine, Law) and three architectures (LLaMA-2-13B, Qwen3-8B, Gemma 4 E4B-it), RLearner-LLM achieved up to a 6x NLI improvement over SFT baselines, with gains in 11 of 15 cells. Gemma 4 E4B-it (4.5B effective parameters) showed NLI increases from +11.9% to +2.4x across four domains with faster inference. Its Qwen3-8B variant won 95% of blind pairwise comparisons against its SFT baseline, but lost 95% to GPT-4o-mini, underscoring the verbosity bias in LLM-as-a-judge evaluations.

Key takeaway

For Machine Learning Engineers developing LLMs for knowledge-intensive applications, recognize that standard DPO approaches can inadvertently sacrifice logical grounding for linguistic fluency due to inherent verbosity bias in preference signals. You should consider implementing a dual-signal Hybrid-DPO framework, like RLearner-LLM, which combines NLI entailment with a verifier score. This method significantly improves logical correctness (up to 6x NLI gains) without degrading fluency, offering a robust path to more accurate and reliable model outputs. Prioritize logic-aware automatic metrics over LLM-as-a-judge for ground truth evaluation.

Key insights

RLearner-LLM resolves DPO's logical alignment gap by fusing NLI and verifier signals, overcoming verbosity bias and alignment tax.

Principles

• Standard DPO preference signals exhibit verbosity bias.
• Single-signal optimization incurs an "alignment tax."
• Logical grounding and linguistic fluency are complementary objectives.

Method

RLearner-LLM uses Hybrid-DPO, an automated pipeline fusing a DeBERTa-v3 NLI entailment score with a min-max-normalized verifier LLM score to construct preference pairs for DPO training.

In practice

• Use NLI and verifier signals for DPO preference.
• Apply Hybrid-DPO to knowledge-intensive tasks.
• Prioritize logic-aware metrics over LLM-as-a-judge.

Topics

• Large Language Models
• Direct Preference Optimization
• Logical Grounding
• NLI Entailment
• Reward Modeling
• LLM Evaluation Bias

Code references

• 14H034160212/Explanation-Generation

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization — Preference alignment, specifically DPO, effectively balances validity and minimality in multilingual counterfactual explanation generation for LLMs.
• Preference Fine-Tuning LFM 2 Using DPO — LFM 2 models combine efficient architecture with DPO fine-tuning for high-performance, edge-deployable SLMs.
• If an LLM could genuinely deduce the intellectual baseline of its user from provided documents or conversational context, it could eliminate boilerplate introductions, redundant explanations, and... — LLMs' lack of true epistemic modeling drives costly verbosity and reduces accuracy, necessitating explicit mitigation strategies.
• Controlled Reformulation Testing for Logical Consistency in Large Language Models — LLMs struggle with logical consistency across equivalent question reformulations despite high base accuracy.
• Token-weighted Direct Preference Optimization with Attention — Using an LLM's intrinsic attention to weight tokens during preference optimization significantly improves alignment and performance.
• Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek R1 enhances LLM reasoning via pure reinforcement learning and rule-based rewards, fostering autonomous problem-solving without supervised data.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.