Know Your Source: A Public Knowledge Store for Media Background Checks

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

MediaRef is a publicly available knowledge store of web-sourced documents covering 200 media outlets, designed to enable reproducible and low-cost evaluation of Media Background Check (MBC) generation. It addresses the limitations of existing LLM-based automated fact-checking (AFC) systems that rely on expensive, proprietary search APIs, which hinder reproducibility and introduce variability. The resource was constructed using a systematic methodology involving targeted web scraping and quality control. Experiments evaluating LLMs like gpt-4o-mini, gpt-5-mini, llama-3.3-70b-instruct, and mistral-7b-instruct demonstrate that using MediaRef improves fact recall in generated MBCs. However, qualitative analysis reveals that producing highly informative and verifiable MBCs remains a significant challenge, particularly for local news sources.

Key takeaway

For research scientists and NLP engineers developing automated fact-checking systems, MediaRef offers a critical, cost-effective resource. You should integrate this publicly available knowledge store to ensure reproducibility and reduce reliance on expensive, variable proprietary search APIs. Apply its qualitative evaluation framework to refine LLM outputs, focusing on generating highly informative and verifiable media background checks, especially for less prominent local news sources. This will enhance the trustworthiness and utility of your AFC models.

Key insights

MediaRef provides a reproducible, low-cost knowledge store for LLM-based media background checks, improving fact recall.

Principles

• Source credibility is vital for RAG systems.
• Decoupling retrieval from generation improves reproducibility.
• MBCs need clarity, relevance, informativeness, verifiability.

Method

MediaRef is built by targeted Google Search API queries for 200 outlets, filtering URLs, then scraping content. MBCs are generated zero-shot, then refined iteratively with BM25-retrieved evidence and DeBERTa-extracted passages.

In practice

• Use MediaRef for reproducible MBC generation.
• Evaluate LLM outputs via clarity, relevance, informativeness, verifiability.
• Address information loss and conflicting data.

Topics

• MediaRef
• Media Background Checks
• Automated Fact-Checking
• Retrieval-Augmented Generation
• LLM Evaluation
• Source Credibility

Code references

• nedjmaou/mediaref
• meta-llama/llama3

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• ContextClaim: A Context-Driven Paradigm for Verifiable Claim Detection — Integrating external context retrieval significantly improves verifiable claim detection by providing relevant background information.
• Beyond Keyword Alerts: Building a Semantic PR Opportunity Monitor for a Real Agency — Semantic retrieval and RAG effectively identify proactive PR opportunities by matching client expertise to news stories.
• # I Built a System That Scores Oil Company Climate Claims Against NGO Evidence. — A RAG pipeline can provide traceable, evidence-backed verdicts for corporate climate claims, overcoming limitations of keyword matching or manual review.
• CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild — CommunityFact offers a dynamic benchmark revealing LLM misinformation detection challenges and web-search benefits.
• RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora — RARE framework improves RAG evaluation by accounting for document redundancy in high-similarity real-world corpora.
• ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents — Accurate source attribution is an independent and critical dimension for factuality verification in MCP-based LLM agents.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.