AI incidents, audits, and the limits of benchmarks

2026-02-13 · Source: Practical AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, AI Safety & Evaluation · Depth: Intermediate, extended

Summary

Sean McGregor, co-founder of the AI Verification & Evaluation Research Institute and founder of the AI Incident Database, discusses AI safety, verification, evaluation, and auditing. He highlights the rapid transition of AI from research to real-world deployment, where incidents have tangible consequences. McGregor explains the AI Incident Database, which has collected over 5,000 human-annotated reports across more than 1,000 discrete incidents, emphasizing its role in motivating safety practices by learning from past failures, akin to aviation or medical adverse event reporting. The discussion also covers the limitations of traditional benchmarks for practical AI applications, the importance of third-party auditing for frontier models, and insights from the DEF CON Generative Red Team Challenge, which revealed the need for statistical rigor in identifying systematic vulnerabilities rather than relying on anecdotal exploits.

Key takeaway

For AI/ML Directors overseeing model deployment, recognize that traditional benchmarks are often insufficient for real-world safety. Your teams should prioritize implementing robust incident reporting mechanisms and consider third-party auditing for critical AI systems to ensure they meet practical safety standards, especially at the interfaces of combined models. This proactive approach will build trust and prevent costly failures, aligning safety with business imperatives.

Key insights

AI safety requires systematic incident reporting and rigorous third-party evaluation to manage real-world risks.

Principles

• Bad things happen, ensure they don't happen again.
• Benchmarks for research differ from practical AI deployment.
• Interfaces between AI systems are often undertested.

Method

The AI Incident Database collects human-annotated reports of AI incidents to identify failure modes and inform safety practices, moving towards mandatory reporting for comprehensive insight into incident rates.

In practice

• Implement mandatory reporting for severe AI incidents.
• Conduct third-party audits for frontier AI models.
• Require statistical evidence for AI system vulnerabilities.

Topics

• AI Safety
• AI Incident Database
• AI Auditing
• Model Evaluation
• Red Teaming

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Machine Learning Engineer, MLOps Engineer, AI Ethicist

Related on AIssential

• Situational Awareness in Government, with UK AISI Chief Scientist Geoffrey Irving — AI's theoretical fragility, combined with rapid capability growth, necessitates robust evaluation and foundational safety research.
• ASMR-Bench: Auditing for Sabotage in ML Research — ASMR-Bench reveals current LLMs and human auditors struggle to detect subtle ML research sabotage.
• Fresh Benchmarks, Reliable Scores: Introducing Continuous Prompt Stewardship for AI Risk Evaluation — Continuous, data-driven prompt stewardship is essential for maintaining the diagnostic power and integrity of AI risk evaluation benchmarks against evolving models.
• AI in audit: The gap between knowing and doing — Audit firms must prioritize AI governance and cultivate auditor skepticism over mere deployment to realize true value.
• Auditing Agent Harness Safety — Agent safety evaluation must audit full execution trajectories, not just final outputs, to detect critical mid-trajectory violations.
• British Police Built a Sprawling Crime-Prediction Machine. Some Results Couldn’t Be Trusted — Predictive policing models demand rigorous, continuous auditing and comprehensive bias testing for reliability and ethical deployment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Practical AI.