Curated retrieval versus open web search in public AI information services: a coverage-trust trade-off

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, AI Governance & Public Services · Depth: Expert, extended

Summary

Evrópuvefur, an independent, government-funded AI service in Iceland, underwent a pre-launch expert evaluation in May-July 2026, ahead of the 29 August 2026 referendum on resuming EU accession talks. The service, run by the University of Iceland, uses Google's Gemini models to answer questions about the European Union, employing either a curated local corpus (RAG) or open web search. Five domain experts conducted 551 evaluations of 449 AI-generated answers. A significant finding revealed that 35% (65 of 187) of web-search answers contained at least one flagged source, primarily for untrustworthiness or irrelevance, while curated sources were flagged less often and only for being outdated. The study identified a coverage-trust trade-off: web search provided broader coverage but compromised source quality, whereas the curated corpus was trustworthy but limited, leading the model to decline answers when information was insufficient. Notably, the system never cited RÚV, Iceland's most trusted news source, and prompt-level steering to prefer trusted domains only weakly increased compliance from 12% to 21%. Fluency and topical fit did not predict source trustworthiness.

Key takeaway

For public sector AI implementers deploying LLM-powered information services, you must prioritize explicit source trustworthiness assessment beyond surface-level answer quality. Recognize that open web search offers broad coverage but introduces significant trust risks, as 35% of web-sourced answers in this study contained flagged untrustworthy material. Conversely, curated knowledge bases ensure trust but limit coverage. Implement robust source auditing and provenance disclosure to users, and consider expanding curated corpora or developing automated source-vetting mechanisms to mitigate the inherent coverage-trust trade-off, especially in high-stakes civic contexts like referenda.

Key insights

Public AI services face a critical coverage-trust trade-off between open web search and curated knowledge bases.

Principles

• Source trustworthiness is a distinct information quality dimension.
• Prompt-level steering weakly governs source selection.
• Answer fluency does not imply source soundness.

Method

Experts evaluated AI answers using a seven-criterion quality rubric and separately flagged individual cited sources for trustworthiness, relevance, or currency.

In practice

• Implement source-trust metrics in routine AI service evaluations.
• Disclose provenance of AI-generated answers to users.
• Consider automated source-trust vetters for live systems.

Topics

• Large Language Models
• Retrieval-Augmented Generation
• Source Trustworthiness
• Information Quality
• Public Information Services
• Expert Evaluation
• Referendum Information

Best for: AI Architect, Research Scientist, CTO, AI Scientist, Policy Maker, AI Ethicist

Related on AIssential

• Yet another experiment proves it's too damn simple to poison large language models — LLMs with web search are vulnerable to simple, low-cost data poisoning via fabricated online sources.
• ProvenAI: Provenance-Native Traces of Evidence in Generated Answers — Meaningful transparency in retrieval-grounded QA requires independently measuring correctness, citation fidelity, and source influence.
• RAND’s report shows that RAG, GraphRAG, and long-context AI systems can appear grounded in trusted documents while still misreading nuance, caveats, evidence strength, and partial truths. — Grounded AI systems can misinterpret nuanced information, even with citations, requiring rigorous, domain-specific evaluation.
• Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant — Human-curated context, not raw schemas, is essential for trustworthy AI data assistants at scale.
• Microsoft's new MAI models — Microsoft's new MAI models leverage MoE architectures and filtered web data, highlighting ongoing data licensing complexities.
• One Polluted Page Is Enough: Evaluating Web Content Pollution in Generative Recommenders — Generative recommenders are highly vulnerable to web content pollution, even from a single source, with current defenses largely ineffective.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.