Machine Rulemaking: Arbitrary and Capricious Review in the Age of AI

2025-05-12 · Source: Law and Technology - Harvard Law Review · Field: Legal & Regulatory — Legal Technology (LegalTech), Regulatory Affairs & Government Relations, Compliance & Risk Management · Depth: Advanced, extended

Summary

Federal agencies are increasingly adopting AI and machine learning (ML) models for various governance functions, including automating tasks and analyzing vast datasets, as evidenced by 157 ML use cases across 64 agencies in 2020. While ML offers potential benefits like recovering $1 billion in check fraud for the U.S. Treasury, it also introduces risks such as reinforcing biases or sudden inaccuracies, especially when applied in critical public sectors like the Social Security Administration (SSA). This analysis focuses on how agency use of ML in rulemaking creates novel issues for the Administrative Procedure Act's (APA) § 706(2)(A) "arbitrary and capricious" review. It examines three hypothetical case studies (SEC, FDA, SSA rules) to demonstrate that while agencies might formally comply with APA requirements by providing "second-order" data on model design and testing, this often fails to achieve the APA's underlying goals of transparency, accountability, and rationality due to ML's inscrutable and evolving nature.

Key takeaway

For CTOs and VPs of Engineering/Data overseeing AI/ML adoption in regulated environments, recognize that formal compliance with existing administrative law by detailing ML model design and testing may not suffice for true transparency and accountability. Your teams should prioritize implementing robust ex post monitoring and reporting mechanisms for ML models in production, especially those impacting critical public services. This proactive approach can mitigate risks of arbitrary outcomes and strengthen the defensibility of ML-driven decisions against future legal challenges, even if it requires advocating for new regulatory frameworks or leveraging existing petition processes.

Key insights

ML's inscrutability and dynamic nature challenge traditional administrative law's ability to ensure transparent and rational agency rulemaking.

Principles

• ML models evolve and can make unintuitive predictions.
• Ex ante review of ML design does not guarantee non-arbitrary outputs.
• Agency expertise in ML differs from subject-matter expertise.

Method

The analysis proposes three hypothetical ML rulemaking case studies (SEC, FDA, SSA) and applies the APA's § 706(2)(A) arbitrary and capricious review to each to identify compliance challenges.

In practice

• Agencies can provide "second-order" data on ML design for formal compliance.
• Consider ex post obligations for ML rules, like ongoing status reports.
• Challenge agency denials of rulemaking petitions for obsolete ML rules.

Topics

• Machine Learning Rulemaking
• Administrative Procedure Act
• Arbitrary and Capricious Review
• AI Governance
• Model Transparency

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Legal Professional, Policy Maker, AI Ethicist

Related on AIssential

• Governance Is Not a Prompt — Existing model risk management frameworks are inadequate for agentic AI, necessitating a new governance paradigm focused on external, enforceable controls.
• Lost in Translation: How AI Exposes the Rift Between Law and Logic — AI-driven data usage demands a structured, machine-readable framework to bridge the gap between legal intent and technical compliance.
• British Police Built a Sprawling Crime-Prediction Machine. Some Results Couldn’t Be Trusted — Predictive policing models demand rigorous, continuous auditing and comprehensive bias testing for reliability and ethical deployment.
• A Critique of Transparency Provisions in NY’s RAISE Act (1.0) — AI governance through transparency faces challenges in defining regulated entities and ensuring disclosed information's accuracy and understandability.
• Governments Are Using AI To Draft Legislation. What Could Possibly Go Wrong? — AI is being adopted by governments worldwide to enhance legislative efficiency, but raises significant concerns about legitimacy and trust.
• The Once And Future Fable #5 — The "Mythos Moment" reveals an ad hoc, reactive US AI policy struggling with frontier model control, legal frameworks, and open-source risks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Law and Technology - Harvard Law Review.