AI as “Arbitrary” Intelligence

2026-06-23 · Source: The Regulatory Review · Field: Legal & Regulatory — Regulatory Affairs & Government Relations, Compliance & Risk Management, Public Policy & Governance · Depth: Intermediate, medium

Summary

Federal agencies' increasing reliance on Artificial Intelligence (AI) for decision-making, with use cases doubling and generative AI increasing nine-fold from 2023 to 2024, poses significant challenges for judicial review under the Administrative Procedure Act's "arbitrary and capricious" standard. The "black box" nature of AI makes it difficult for judges to determine if agencies rely on "improper factors" or fail to consider important aspects, as demonstrated by the Arkansas Department of Health Services' algorithm that negatively affected nearly half of Medicaid recipients. Furthermore, the common practice of procuring third-party AI, where vendors often shield training data, introduces risks that models incorporate problematic content from sources like social media, leading to decisions based on constitutionally protected characteristics such as race or gender, as seen in Amazon's biased recruiting tool. Existing judicial workarounds for human decision-makers are insufficient for AI due to the absence of human imperfection and institutional safeguards.

Key takeaway

For policy makers and legal professionals evaluating AI integration into federal agencies, recognize that current administrative law frameworks are ill-equipped for AI's "black box" and opaque third-party training data. Your agency risks judicial invalidation under the Administrative Procedure Act if AI decisions rely on improper factors or violate Equal Protection. Mandate transparency in AI procurement, especially for training data, to ensure accountability and prevent arbitrary or biased outcomes.

Key insights

AI's "black box" and third-party data complicate judicial review of agency decisions, risking arbitrary outcomes and constitutional violations.

Principles

• Agencies must avoid "improper factors."
• Agencies cannot ignore "important aspects."
• Outsourced AI data integrity is critical.

Topics

• Judicial Review
• Administrative Procedure Act
• AI Governance
• Algorithmic Bias
• Third-Party Data
• Federal Agencies

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Legal Professional, Policy Maker, AI Ethicist

Related on AIssential

• After trial judge cites AI-generated judgment, Supreme Court asks BCI to set up expert panel to examine issue - Bar and Bench — AI-generated legal "hallucinations" necessitate judicial accountability and sovereign AI development in India.
• CLAUDE JUST GOT BANNED — A U.S. government directive has banned Anthropic's AI due to a standoff over military usage terms, impacting its market position.
• Police AI chief admits crime-fighting tech will have bias but vows to tackle it | AI (artificial intelligence) - The Guardian — Police AI will contain bias, necessitating robust mitigation strategies and human oversight for fair outcomes.
• Kleiner v. Adobe is another step in a pattern: the legal system is increasingly treating “training data governance” as a compliance domain, not a research footnote. — AI training data governance is evolving into a critical legal compliance domain, not merely a research consideration.
• US justice department steps in on behalf of xAI in Colorado regulation case — Federal intervention in xAI's lawsuit against Colorado's AI law escalates the debate over state versus federal AI regulation.
• The Legal Questions AI Is Forcing Every Agency to Face — Agencies must proactively address AI's legal risks, including IP and data privacy, to innovate responsibly.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Regulatory Review.