There should be $100M grants to automate AI safety

2026-04-03 · Source: AI Alignment Forum · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, long

Summary

An editorial analyst proposes that AI safety funders should establish "automated AI safety scaling grants" of up to $100 million to aggressively incentivize the use of automated AI labor for safety work. The motivation stems from a "short timeline" perspective, suggesting current funding is too conservative and insufficient. The proposed grant structure involves a multi-stage process: an initial $5M grant to demonstrate scalability, followed by a $14M "double up" grant upon meeting scaling conditions, and then a substantial $108M "big scale" grant, with $100M specifically for pipeline scaling. Successful projects would require public benefit, such as open-sourcing the pipeline or collaborating with AGI labs. Potential areas for these grants include monitoring & control, automated black box auditing (improving on systems like PETRI), white box auditing agents, propensity evaluations, and automated conceptual alignment research, despite concerns about goodharting metrics.

Key takeaway

For research scientists focused on AI safety, consider developing scalable, automated pipelines that can demonstrate clear safety improvements. Your proposals should outline empirical evidence for scalability and a plan for public benefit, as funders are being urged to offer substantial grants (up to $100M) for projects that can rapidly scale automated safety solutions, moving beyond traditional funding conservatism.

Key insights

Funders should offer large grants to scale automated AI labor for safety, given short timelines and conservative past funding.

Principles

• Aggressively incentivize automated AI safety.
• Prioritize scalable safety pipelines.
• Ensure public benefit from funded research.

Method

A multi-stage grant process: initial funding to show scalability, followed by rapid, significantly larger grants (up to $100M+) to scale successful automated AI safety pipelines, with public benefit as a condition.

In practice

• Develop scalable monitoring & control systems.
• Improve automated black box auditing tools.
• Create automated propensity evaluation methods.

Topics

• AI Safety Funding
• Automated AI Safety
• Scaling Grants
• AI Monitoring
• Black Box Auditing

Best for: Research Scientist, AI Scientist, Director of AI/ML, Entrepreneur

Related on AIssential

• Open Thread 428 — AI safety and biosecurity funding opportunities are expanding, alongside community updates and niche service advertisements.
• xAI says it raised $20B in Series E funding — xAI secured $20 billion in Series E funding amid international investigations into Grok's generation of illicit content.
• Guiding a Safe Future for AI – Part 1 — AI's automation of intelligence demands collaborative, multi-faceted safety approaches across immediate, societal, catastrophic, and superintelligence risks.
• Advancing independent research on AI alignment — Independent research is crucial for AI alignment, expanding ideas beyond frontier labs.
• Building & Scaling the AI Safety Research Community, with Ryan Kidd of MATS — AI safety requires a portfolio approach, balancing technical research with governance and preparing for uncertain AGI timelines.
• Responses to the AI grant flood must prioritize fairness as part of excellence — AI-driven application surges demand grant system reform to ensure fairness and maintain funder credibility.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Alignment Forum.