Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

2026-07-07 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Large Reasoning Models (LRMs) enhance language model capabilities by generating explicit thinking traces for factuality-oriented question answering (QA). While often improving performance by recovering knowledge, this "thinking" can also cause "thinking-induced hallucination," where correct direct answers are overturned, leading to factual drift. Researchers explain this as a thinking residual that can either recover knowledge or introduce unsupported associations. To address this, they propose MARGO, Mixed-Mode Advantage Regularization for Grounded Optimization. MARGO is a reinforcement learning framework that utilizes non-thinking rollouts as same-model references for advantage estimation. By creating mixed-mode rollout groups with both thinking and non-thinking trajectories, MARGO assesses whether explicit thinking adds factual value beyond direct answering. This approach effectively suppresses hallucination-prone thinking while retaining beneficial reasoning behaviors. Experiments on multiple factuality-oriented QA benchmarks demonstrate MARGO's improved factual reliability over strong baselines, and evaluations on mathematical benchmarks confirm it preserves general reasoning ability.

Key takeaway

For AI Scientists and Machine Learning Engineers developing Large Reasoning Models for factuality-oriented QA, you should consider integrating MARGO to enhance factual reliability. This reinforcement learning framework helps mitigate "thinking-induced hallucination" by evaluating the factual value of explicit thinking against direct answers. It ensures your models maintain accuracy without sacrificing general reasoning capabilities on complex tasks. Implement MARGO to suppress detrimental thinking while preserving beneficial reasoning.

Key insights

Explicit thinking in LRMs can induce factual hallucinations; MARGO mitigates this by regularizing thinking based on non-thinking references.

Principles

• Explicit thinking can introduce factual drift.
• Evaluate thinking's factual value against direct answers.
• Suppress harmful thinking, preserve beneficial reasoning.

Method

MARGO is an RL framework using non-thinking rollouts as same-model references in advantage estimation. It constructs mixed-mode rollout groups to evaluate thinking's factual value, suppressing hallucination-prone thinking.

In practice

• Apply MARGO to improve LRM factual reliability.
• Use mixed-mode rollouts for hallucination suppression.
• Preserve general reasoning in mathematical tasks.

Topics

• Large Reasoning Models
• Factual Hallucination
• Question Answering
• Reinforcement Learning
• MARGO
• Model Reliability

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Thinking Past the Answer: Evaluating Harmful Overthinking in Large Reasoning Models — Large Reasoning Models often degrade performance by continuing to reason past an already-correct answer.
• MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucination — MARCH uses multi-agent reinforcement learning and information asymmetry to reduce LLM hallucination in RAG systems.
• Thinking to recall: How reasoning unlocks parametric knowledge in LLMs — Reasoning in LLMs unlocks parametric knowledge through latent computation and factual priming, even for simple facts.
• Shameless Guesses, Not Hallucinations — AI "hallucinations" are better understood as "shameless guesses" stemming from their reward-driven training process.
• Expert Masking for More Efficient Expert Offloading — Fine-tuning can degrade existing factual knowledge, but strategies like parameter restriction or self-distillation can mitigate this.
• Pseudocode-Guided Structured Reasoning for Automating Reliable Inference in Vision-Language Models — PStar uses pseudocode and adaptive strategy selection to reduce VLM hallucinations for safer robotic automation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.