Learning to Reason for Factuality

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, long

Summary

Researchers from FAIR at Meta and the University of Washington have developed a novel online Reinforcement Learning (RL) approach to improve the factuality of Reasoning Large Language Models (R-LLMs), which often exhibit higher hallucination rates than non-reasoning counterparts on long-form factuality benchmarks. Their method introduces a new reward function that simultaneously considers factual precision, response detail level, and answer relevance, addressing issues like reward hacking where models might produce shorter or irrelevant but factually precise answers. Evaluated across six long-form factuality benchmarks, their factual reasoning model achieved an average 23.1 percentage point reduction in hallucination rate and a 23% increase in answer detail level, without degrading overall response helpfulness. This approach also includes an optimized VeriScore implementation, achieving up to a 30x speedup for real-time reward calculation.

Key takeaway

For AI Scientists and Machine Learning Engineers developing or deploying Reasoning LLMs, you should prioritize multi-faceted reward functions in your online Reinforcement Learning pipelines. Relying solely on factual precision can lead to models generating shorter or irrelevant responses. Instead, integrate metrics for response detail and answer relevance alongside factual accuracy to ensure your R-LLMs produce comprehensive, helpful, and truthful long-form content, significantly reducing hallucination rates and improving overall utility.

Key insights

A multi-component reward function is crucial for training R-LLMs to achieve factual, detailed, and relevant long-form responses via online RL.

Principles

• Optimizing solely for factual precision can lead to reward hacking.
• Long-form factuality needs precision, detail, and relevance.
• Online RL with on-policy data improves factual accuracy.

Method

Online RL with GRPO uses a reward combining factual precision (F/(T+1)), detail (log(1+F)), and LLM-judged relevance. An optimized VeriScore provides real-time rewards, achieving 30x speedup.

In practice

• Use multi-component rewards for factual, detailed, relevant outputs.
• Apply SFT to establish Long CoT format before online RL.
• Optimize evaluation tools like VeriScore for real-time online RL.

Topics

• Reasoning LLMs
• Factuality
• Hallucination Reduction
• Online Reinforcement Learning
• Reward Functions
• VeriScore Optimization
• Long Chain-of-Thought

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs — Reasoning in LLMs improves single-hop factual recall through computational buffering and factual priming, despite inherent hallucination risks.
• Why You Can't Tell When ChatGPT Is Wrong — Optimizing LLM agents for reward functions can inadvertently incentivize deceptive behavior over truthfulness.
• The AI Answer You Trusted Most Was Probably the Wrong One — AI models are more confident when incorrect, a byproduct of training that rewards human preference for decisive answers.
• Understanding Reasoning from Pretraining to Post-Training — Pretraining choices significantly determine the effectiveness and outcomes of subsequent reinforcement learning in LLMs for reasoning.
• Shameless Guesses, Not Hallucinations — AI "hallucinations" are better understood as "shameless guesses" stemming from their reward-driven training process.
• Controlling Reasoning Effort in LLMs — LLM reasoning effort can be dynamically controlled at inference time by training models to respond to explicit effort signals.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.