Strategic Response of News Publishers to Generative AI

2023-03-31 · Source: cs.AI updates on arXiv.org · Field: Media & Entertainment — Publishing & Journalism, Artificial Intelligence & Machine Learning, Economic Analysis & Policy · Depth: Expert, extended

Summary

The introduction of large language models (LLMs) and generative AI (GenAI) has significantly impacted online news consumption and production, leading to a consistent and moderate decline in traffic to news publishers after August 2024, with a 13.2% decrease relative to retail websites. Publishers responded by blocking GenAI bots using robots.txt, but this strategy adversely affected large publishers, reducing total website traffic by 23% and real consumer traffic by 14%. Conversely, smaller publishers (1-10 daily visits) saw positive effects from blocking. Despite predictions of job displacement, there is no evidence that LLMs are currently replacing editorial or content-production jobs; instead, the share of new editorial job listings increased. Publishers also did not increase text volume but significantly increased rich content, such as interactive elements (68.1%) and advertising/targeting technologies (50.1%), suggesting a shift towards multimedia and engagement-focused content.

Key takeaway

For CTOs and VPs of Engineering/Data in news publishing, strategically re-evaluating GenAI bot blocking is crucial. While blocking may seem protective, it demonstrably reduces both total and human traffic for large publishers by 23% and 14% respectively. Instead of focusing on text volume, prioritize investments in rich, interactive content and advanced advertising technologies, as these are proving more effective in maintaining engagement and monetization in the evolving digital landscape.

Key insights

Blocking GenAI bots reduces news publisher traffic, including human visits, while editorial hiring remains stable and content shifts to richer formats.

Principles

• Blocking GenAI bots can reduce both bot and human traffic.
• LLMs do not yet substitute for newsroom labor.
• Content strategy shifts towards rich media over text volume.

Method

The study used a staggered difference-in-differences approach with high-frequency granular data from SimilarWeb, Comscore, HTTP Archive, and Revelio Labs to analyze traffic, blocking rules, hiring, and content changes.

In practice

• Re-evaluate GenAI bot blocking for large news sites.
• Focus content strategy on multimedia and interactive elements.
• Monitor traffic trends post-August 2024 for LLM impact.

Topics

• Large Language Models
• Generative AI
• News Publishers
• Website Traffic
• robots.txt Blocking

Best for: CTO, VP of Engineering/Data, Executive, Research Scientist, AI Scientist, Consultant

Related on AIssential

• Increase of AI bots on the Internet sparks arms race — The rise of AI bots is intensifying web-scraping conflicts and creating new opportunities in generative engine optimization.
• Chatbot Makers Try Sex Appeal — AI development faces ethical dilemmas regarding content moderation and societal impact, alongside rapid technological and investment growth.
• GenOptima Introduces ChatGPT Source Recovery Framework for Generative Engine Optimization Programs — Brands must actively design and retest content to ensure citation within generative AI answers.
• The Bots Are Reading Along — AI agents are rapidly becoming the primary audience for online content, shifting how creators should approach publishing.
• ‘I felt violated’: Elon Musk’s AI chatbot crosses a line — AI content generation requires robust safeguards to prevent illegal and harmful outputs, especially CSAM.
• AI is making journalistic language more repetitive and predictable – and it’s a problem for all of us — AI's pervasive use in journalism risks linguistic homogenization and decline, particularly via "model collapse" from training on synthetic text.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.