Amnesty: many mainstream standalone generative AI systems are not merely risky in deployment, but structurally incompatible with international human rights law because of how they are built.

2025-11-28 · Source: Pascal’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Regulatory Affairs & Government Relations, Social Sciences & Behavioral Studies · Depth: Intermediate, long

Summary

Amnesty International's report, "Unlawful by Design: Exposing the Human Rights Costs of Generative AI," asserts that many mainstream standalone generative AI systems are structurally incompatible with international human rights law due to their design, not just their deployment risks. The report argues that harms, including violations of privacy, equality, freedom of expression, thought, children's rights, environmental justice, and democratic accountability, stem from design choices like mass web scraping and opaque training. It draws an analogy between generative AI training and mass surveillance, highlights concerns about AI's impact on autonomous thought formation, and emphasizes the environmental costs and cultural dominance of Anglophone data. The report also addresses the foreseeable generation of child sexual abuse material and technology-facilitated gender-based violence.

Key takeaway

For policy makers developing AI governance frameworks, you must move beyond voluntary ethics and focus on the entire AI supply chain. Require mandatory human rights and environmental impact assessments before deployment, not after harm occurs. Your regulations should enforce data provenance, transparency for training data, and effective remedies for affected individuals. Consider bright-line prohibitions for systems built on non-consensual data extraction, especially sensitive personal or children's data.

Key insights

Generative AI's core design, particularly mass data extraction, fundamentally conflicts with international human rights law.

Principles

• Generative AI harms are design choices, not accidental glitches.
• Legality must be assessed at the AI design stage, not just deployment.
• Model scale does not inherently equate to better or more universal intelligence.

In practice

• Develop small, domain-specific models with curated datasets.
• Implement provenance-preserving architectures for training data.
• Prioritize local-language investment and human-in-the-loop governance.

Topics

• Generative AI
• Human Rights Law
• AI Governance
• Data Provenance
• Environmental Justice
• Child Safety

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Policy Maker, AI Ethicist, Legal Professional

Related on AIssential

• The Urgency of Standards for Synthetic Data in the Era of Agentic AI — Unregulated synthetic data poses significant risks to agentic AI systems, necessitating urgent standardization and policy adaptation.
• AI and the Dangerous Fiction of ‘All Lawful Use’ — The "all lawful use" standard for government AI contracts risks human rights violations due to misaligned national laws and global scalability issues.
• The AI Ethics Brief #188: The Names We Give Things — The gap between AI's stated purpose and its actual implementation creates ethical and privacy challenges.
• India Orders X To Control Grok After Obscene AI Content Surfaces — Regulatory bodies are actively intervening to control AI-generated content deemed unlawful or harmful.
• Is AI already killing people by accident? — Generative AI's unreliability and moral hazard in military targeting demand urgent scrutiny and accountability.
• The Guardian view on changes to copyright laws: authors should be protected over big tech | Editorial — Authors are demanding robust copyright legislation to protect their work and livelihoods from unlicensed AI use.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Pascal’s Substack.