LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

LongCrafter is a novel structured synthesis framework designed to enhance large language models' (LLMs) long-context understanding by addressing limitations in existing supervised fine-tuning (SFT) data: narrow task coverage, insufficient instruction difficulty, and lack of faithfulness supervision. The framework employs a hierarchical task taxonomy, organizing long-context understanding into 32 fine-grained task types across local/shallow and global/deep levels. LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs modeling cross-paragraph dependencies, and generates instruction-response pairs strictly grounded in located evidence spans. Models fine-tuned on LongCrafter's 2,000 samples consistently outperform SFT baselines and official post-trained models on benchmarks like LongBench, LongBench v2, and LooGLE, achieving All-Overall scores of 45.15% on Qwen2.5-7B and 45.71% on LLaMA-3.1-8B, with gains of +2.41 and +5.25 points respectively. This approach also effectively mitigates the "lost in the middle" problem by improving positional robustness in evidence localization.

Key takeaway

For AI Scientists and Machine Learning Engineers developing long-context LLMs, you should consider integrating structured data synthesis methods like LongCrafter. This approach, which uses evidence graphs and a comprehensive task taxonomy, significantly improves model performance on complex, multi-hop reasoning tasks and mitigates the "lost in the middle" problem. Implementing evidence-grounded instruction synthesis can lead to more robust and faithful long-context understanding, even with a modest 2,000 training samples.

Key insights

LongCrafter improves LLM long-context understanding via evidence-graph-guided instruction synthesis, ensuring diverse, difficult, and faithful training data.

Principles

• Hierarchical task taxonomies guide diverse data synthesis.
• Evidence graphs model cross-paragraph dependencies.
• Citation-grounded responses ensure faithful reasoning.

Method

LongCrafter's three-stage pipeline involves Long Context Construction, Evidence-Constraint Graph Construction, and Instruction-Response Pair Synthesis, all guided by a 32-type task taxonomy.

In practice

• Use evidence graphs for multi-hop QA data.
• Implement step-by-step citation in responses.
• Design tasks with varying evidence locality.

Topics

• Long-Context LLMs
• Supervised Fine-tuning
• Evidence Graphs
• Instruction Synthesis
• Task Taxonomy
• Data Diversity
• Positional Robustness

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation — LLMs struggle to consistently translate improved latent inference into rational downstream predictions in multi-turn Bayesian tasks.
• DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning — LLMs excel at orchestrating graph algorithms via tools but struggle with compositional multi-hop graph reasoning.
• Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning — A data-centric approach with a specific recipe and minimal GRPO setup substantially improves long-context reasoning in LLMs.
• Longer Context Silently Shortens LLM Reasoning — Context length and management significantly impact LLM reasoning efficiency and accuracy.
• ACC: Compiling Agent Trajectories for Long-Context Training — Agent Context Compilation (ACC) transforms multi-turn agent trajectories into long-context QA pairs, directly supervising long-range reasoning.
• Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment — An agentic LLM pipeline scalably labels architectural quality, significantly improving code LLM reasoning and patch conformance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.