The emergence of the web data infrastructure layer for AI

2026-06-24 · Source: MIT Technology Review · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Data Science & Analytics · Depth: Intermediate, medium

Summary

The emergence of a new web data infrastructure layer is critical for AI, as current models struggle with the dynamic, unstructured nature of web data. The web was not designed for the automated discovery and retrieval AI applications demand, leading to issues like AI hallucinations and project abandonment; Gartner reports 60% of AI projects without "AI-ready" data will fail. Bright Data's CEO, Or Lenchner, highlights the need for infrastructure that can mimic human browsing behavior at scale, navigating hundreds of millions of domains and billions of new URLs weekly, delivering real-time information while overcoming technical barriers. This specialized infrastructure, which can emulate a web user with 1,000+ parameters 80 billion times a day, is essential for applications like dynamic pricing and trademark tracking. A survey found 56% of AI practitioners need real-time web data to improve trust, and 97% of AI organizations depend on such infrastructure, though 90% feel restricted.

Key takeaway

For AI Architects or MLOps Engineers building AI systems requiring current, reliable data, you must prioritize specialized web data infrastructure. Your models need real-time, trustworthy information to avoid stale answers and reduce hallucinations. Invest in platforms designed for large-scale, low-latency data retrieval and orchestration, ensuring compliance with privacy frameworks like GDPR and CCPA. This commitment will position your organization to build more responsive and reliable AI systems.

Key insights

Specialized web data infrastructure is critical for AI to access real-time, trustworthy, and contextually relevant information at scale.

Principles

• AI performance depends on a system's compute, networking, retrieval, and data engineering capabilities.
• Static training data is insufficient for AI models operating in dynamic business environments.
• Live, high-quality web data reduces AI hallucinations and builds user trust in model outputs.

Method

A web data infrastructure platform emulates human browsing behavior, accessing content from JavaScript-heavy sites and those with antibot software, mimicking a web user with identifying information (IP, location, 1,000+ parameters) at scale.

In practice

• Implement dynamic pricing engines using public web information.
• Track trademark infringements across global brands.
• Integrate public web retrieval with APIs and proprietary data for AI applications.

Topics

• Web Data Infrastructure
• Real-time Data Retrieval
• AI Data Quality
• Retrieval-Augmented Generation
• Data Governance
• Bright Data

Best for: CTO, VP of Engineering/Data, AI Architect, MLOps Engineer, Director of AI/ML

Related on AIssential

• Semantic Layers in the Wild: Lessons from Early Adopters — Semantic layers are becoming critical AI infrastructure, driven by the need for accurate, governed data for AI analytics.
• The 7 Database Types Powering Every Modern AI System — Successful AI systems depend on orchestrating multiple specialized database types, not just advanced models.
• How Modern Data Pipelines Drive AI Application Development in Dubai — Modern data pipelines are foundational for scalable, real-time AI applications, ensuring data quality and efficient processing.
• Dell Technologies: Is Your Infrastructure AI-Ready? — Purpose-built infrastructure is critical for unlocking AI's full potential and overcoming limitations of legacy systems.
• The Contextual Revolution: The Rise of the Metadata Engineer in the Intelligence Economy — Metadata engineering is crucial for building trusted, context-rich data foundations essential for advanced AI and mitigating hallucinations.
• Your AI Strategy Isn’t Failing Because of the Model — It’s the Data — Data governance and quality, not AI models, are the primary bottlenecks preventing enterprise AI from scaling to production.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MIT Technology Review.