ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents

2026-06-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

ProvenanceGuard is a novel source-aware verifier for LLM agents utilizing the Model Context Protocol (MCP) to synthesize answers from heterogeneous evidence sources. It specifically targets cross-source conflation, a critical failure mode where claims are factually supported but incorrectly attributed to a source. The system operates by consuming MCP traces, decomposing answers into atomic claims, routing these claims to their specific evidence, and verifying support using NLI and token-alignment. It then compares the agent's stated attribution against the routed source, providing per-claim verdicts and an overall allow/block decision. Evaluated on 281 medical-domain MCP-agent traces, ProvenanceGuard achieved a block F1 of 0.802 and source accuracy of 0.858 on a 40-trace held-out split, outperforming source-blind baselines. It successfully detected all injected attribution swaps in 50 clinical conflation probes, demonstrating that source attribution is an independent axis for factuality verification.

Key takeaway

For NLP Engineers and Research Scientists developing or deploying MCP-based LLM agents, you must move beyond pooled-evidence factuality checks. This research demonstrates that source attribution is an independent and critical dimension for verifying agent outputs. You should integrate source-aware verification mechanisms, like ProvenanceGuard, into your agent pipelines to detect and mitigate cross-source conflation. This will significantly improve the reliability and trustworthiness of your agents' responses, especially when dealing with sensitive or heterogeneous data sources, by ensuring claims are not only supported but correctly attributed.

Key insights

Accurate source attribution is an independent and critical dimension for factuality verification in MCP-based LLM agents.

Principles

• Cross-source conflation is a distinct factuality failure mode.
• Source-aware verification improves LLM agent reliability.
• Exact source ownership is difficult with semantically close sources.

Method

ProvenanceGuard decomposes answers into atomic claims, routes them to source-specific evidence, checks support via NLI and token-alignment, compares stated attribution with routed sources, and returns per-claim verdicts for an allow/block decision.

In practice

• Implement source-aware verification in MCP-based agents.
• Use retrieval-augmented revision for blocked answers.

Topics

• LLM Agents
• Factuality Verification
• Model Context Protocol
• Source Attribution
• Cross-source Conflation
• Natural Language Inference
• Medical Domain AI

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval — Fact-checking without retrieval leverages LLM internal representations for robust, scalable claim verification.
• The Provenance Paradox in Multi-Agent LLM Routing: Delegation Contracts and Attested Identity in LDP — Unverified self-claimed quality in multi-agent LLM routing leads to systematically selecting the worst delegates.
• Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation — LLMs can detect fabricated statistics but fail to apply this capability during multi-source synthesis, prioritizing stylistic credibility.
• Verify Before You Commit: Towards Faithful Reasoning in LLM Agents via Self-Auditing — SAVeR improves LLM agent faithfulness by verifying internal beliefs before action, preventing propagation of unsound reasoning.
• Trust Me. I’m Artificial Intelligence. — AI models are becoming more convincing in their reasoning, but this does not equate to increased truthfulness or correctness.
• Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications — Semantic mismatches between LLM tool descriptions and code, termed DCI, pose critical reliability and security threats.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.