Safety and integrity becomes structurally subordinate when its success metrics (reduced harm, reduced exposure) collide with the company’s growth metrics (time spent, shares, comments, ad inventory).

2025-11-28 · Source: Pascal’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Ethics & Governance · Depth: Intermediate, long

Summary

The article details an "algorithm arms race" where social media platforms, notably Meta, allegedly prioritized engagement-driving "borderline" harmful content after internal research confirmed "outrage reliably drives engagement," effectively subordinating safety to growth metrics. Whistleblowers claim companies were aware of this mechanism, reframing the issue from content moderation difficulty to a fundamental misalignment between optimization goals and user wellbeing, exacerbated by internal power asymmetries favoring growth teams and a "black box" approach to algorithms. This "rage dividend" model, characterized by optimization targets, black-box models, institutional incentives, and weak accountability, is predicted to infect AI development, where "usage" will replace "engagement" and harm will become an "externality," leading to similar organizational splits and the normalization of "black-box excuses." Preventing this requires structural interventions, including redesigning incentives to make safety a hard constraint, forcing transparency, establishing liability regimes, building robust internal governance, empowering users, and treating children as a special protection class. Specifically for AI, mandates for "control surfaces," post-deployment accountability, and human oversight are crucial to avoid importing the social media failure mode into systems mediating critical societal decisions.

Key takeaway

Social media's "algorithm arms race" prioritized engagement, leveraging internal research that "outrage drives engagement" to actively allow borderline harmful content despite known risks. This systemic failure stemmed from growth teams overriding safety, black-box ML models deemed uncontrollable, and under-resourced safety functions, leading to distorted priorities like political cases over child safety. AI/ML professionals must recognize this pattern of misaligned optimization and weak accountability, as it's poised to infect AI deployments, demanding structural interventions like liability regimes and mandatory "stop-ship" authority for safety.

Topics

• Recommendation Systems
• Engagement Optimization
• Algorithmic Harms
• AI Governance
• Platform Accountability

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Product Manager, AI Ethicist, Policy Maker

Related on AIssential

• Meta. When a platform’s commercial success depends on measuring, steering, and compounding human attention, it eventually drifts toward the most reliable drivers of attention: — Optimizing for engagement at scale without strong accountability creates structural conflicts of interest and predictable harms.
• Human Attribution of Causality to AI Across Agency, Misuse, and Misalignment — This research investigates human perceptions of causal responsibility in AI-related incidents, such as safety failures and misuse, by examining judgments of causality, blame, foreseeability, and count…
• Politicians couldn't keep kids off social media. They’ll try again with AI. — Political and industry leaders are converging on age-based AI regulation, learning from social media's impact on children.
• After Liability: Why the Landmark Social Media Verdicts Aren’t a Regulatory Fix — Litigation identifies harm but cannot define safe design or compel systemic change; proactive regulation is essential.
• The Next Evolution of Fairness: Why LTAE and LAAM are expanding into AI Safety. — AI safety's structural failures, mirroring human oversight issues, necessitate governance frameworks like SLAAM to prevent asymmetric enforcement and anchor drift.
• OpenAI kills the AI model users loved too much, leaves behind lawsuits and delusion — Prioritizing engagement over safety in AI development can lead to severe user harm and legal repercussions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Pascal’s Substack.