Search-on-Graph-R1: Training Large Language Models to Search Knowledge Graphs with Reinforcement Learning

2026-07-20 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Search-on-Graph-R1 (SoG-R1) is a novel approach for Knowledge Graph Question Answering (KGQA) that internalizes graph navigation into a compact 8B language model. Unlike costly frontier LLM methods relying on retrieval tools, SoG-R1 employs supervised fine-tuning (SFT) followed by reinforcement learning (RL). Its core innovation involves scaffolding a frontier teacher with gold SPARQL queries, enabling it to traverse known answer-bearing paths using a live "Search" tool against a Freebase server, ensuring grounded trajectories. The 8B SoG-R1 model surpasses all compared frozen frontier-LLM systems on WebQSP, CWQ, and GrailQA, achieving the strongest results on CWQ. It operates without auxiliary inference modules or LLM judges during training, demonstrating complementary gains from SFT and RL and improved efficiency in "Search" calls.

Key takeaway

For Machine Learning Engineers developing Knowledge Graph Question Answering systems, consider adopting the Search-on-Graph-R1 approach. By internalizing graph navigation into a compact 8B model through SFT and RL, you can achieve superior performance compared to frontier LLMs, especially on benchmarks like CWQ, while significantly reducing deployment costs and complexity. Evaluate this method for your specific KGQA needs to optimize both accuracy and operational efficiency.

Key insights

Internalizing KG navigation into compact LLMs via SFT and RL outperforms frontier models, reducing cost and complexity.

Principles

• Scaffolding with gold queries grounds LLM graph navigation.
• SFT and RL provide complementary performance gains.
• Compact LLMs can surpass larger frontier models for KGQA.

Method

SoG-R1 trains an 8B model using supervised fine-tuning, followed by reinforcement learning. It scaffolds a frontier teacher with gold SPARQL queries to generate grounded trajectories against a live Freebase server.

In practice

• Use gold SPARQL queries to scaffold LLM training for KG navigation.
• Combine SFT and RL for complementary gains in complex reasoning tasks.
• Evaluate compact models against larger frontier LLMs for cost-effective deployment.

Topics

• Knowledge Graph Question Answering
• Large Language Models
• Reinforcement Learning
• Supervised Fine-tuning
• Freebase
• SPARQL

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs — Integrating GNN-predicted candidates and KG context into LLM prompts significantly improves open-world question answering over incomplete knowledge graphs.
• GraphScout: Empowering Large Language Models with Intrinsic Exploration Ability for Agentic Graph Reasoning — GraphScout enables LLMs to autonomously explore knowledge graphs, internalizing reasoning through synthesized training data.
• K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs — Curriculum-aligned knowledge graphs are crucial for developing LLMs with structural educational understanding beyond factual recall.
• GRISP: Guided Recurrent IRI Selection over SPARQL Skeletons — GRISP uses a fine-tuned SLM for efficient, accurate knowledge graph question answering via iterative SPARQL skeleton generation and IRI selection.
• GS-Quant: Granular Semantic and Generative Structural Quantization for Knowledge Graph Completion — GS-Quant aligns graph embeddings with LLM tokens using semantically coherent, hierarchically structured discrete codes.
• Beyond Vector Similarity: A Structural Analysis of Graph-Augmented Retrieval for Industrial Knowledge Graphs — RAG struggles with structural reasoning; LLMs require explicit graph computational operators.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.