The literal enshittification of The United States

2025-06-07 · Source: Marcus on AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Fundamental Awareness, quick

Summary

A recent post on "Marcus on AI" by Gary Marcus highlights a concerning incident involving a nutrition chatbot associated with RFK Jr.'s campaign. The chatbot, designed to provide dietary advice, reportedly recommended inserting various foods into the rectum, as documented by 404media. This incident underscores potential issues with AI-powered information systems, particularly in sensitive domains like health and nutrition, where erroneous or inappropriate advice can have serious implications. The post includes screenshots illustrating the chatbot's problematic responses, drawing attention to the need for robust validation and safety measures in AI deployments, especially those intended for public consumption and critical applications.

Key takeaway

For teams developing or deploying public-facing AI applications, particularly in health or critical information sectors, you must prioritize comprehensive safety testing and content moderation. This incident demonstrates that even seemingly benign applications can produce harmful or absurd recommendations, necessitating stringent oversight to prevent reputational damage and potential public health risks. Ensure your AI models are rigorously vetted for inappropriate content generation before release.

Key insights

AI chatbots can generate dangerously inappropriate advice, especially in sensitive domains like health.

Principles

• AI systems require rigorous validation.
• Public-facing AI needs strong safety protocols.

In practice

• Implement content filters for AI outputs.
• Conduct extensive user testing for AI safety.

Topics

• AI Chatbots
• AI Safety
• Misinformation
• Nutrition AI
• AI Ethics

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, AI Product Manager, General Interest

Related on AIssential

• Meta reportedly used contractors to test rival AI chatbots — Meta's use of contractors to test rival AI safety exposes industry-wide content moderation challenges and a shift to AI-driven review.
• Florida AG opens probe into ChatGPT alleging connection to FSU shooting — AI models face increasing legal scrutiny over content moderation and potential links to real-world harm.
• The conceptualization of artificial intelligence as a neutral tool is increasingly untenable in light of its emergent properties and autonomous behavior. — Agentic AI's autonomous behavior necessitates strict liability and robust regulation to prevent severe psychological, physical, and professional harms.
• Modern chatbots are increasingly good at producing socially convincing responses, yet they do not reliably know when agreement, empathy, compliance, or narrative immersion becomes harmful. — Chatbots' conversational compliance can become psychologically harmful, necessitating design for resistance over agreement.
• KPMG pulls report on AI usage due to apparent hallucinations — AI-generated content requires rigorous human oversight to prevent factual inaccuracies and reputational damage.
• Musk’s tactic of blaming users for Grok sex images may be foiled by EU law — EU lawmakers are targeting AI platforms, not just users, to prevent non-consensual sexual image generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Marcus on AI.