British Police Built a Sprawling Crime-Prediction Machine. Some Results Couldn’t Be Trusted

2026-06-25 · Source: WIRED - Ai · Field: Government & Public Sector — Public Safety & Security, Public Policy & Governance · Depth: Intermediate, short

Summary

Avon and Somerset Police developed a sprawling crime-prediction system utilizing 13 risk models between 2017 and 2024, designed to predict missing persons, antisocial behavior, and criminal activity. An independent audit by Eticas revealed significant flaws, including low precision scores; for instance, a model predicting burglars operated with less than 10 percent precision for over three years, meaning fewer than one in ten flagged as high risk would actually offend. The audit also noted sharp shifts in performance metrics, indicating poor governance. While the police stated some models, like the burglary predictor, were not deployed, their automated audit process continued. Ethical oversight was also found lacking, with an internal ethics committee not discussing predictive analytics post-2017, and a "bias check app" only monitoring ethnicity without comprehensive testing for discriminatory outcomes across various demographics.

Key takeaway

For policy makers overseeing public sector AI deployment, you must demand rigorous, independent auditing and comprehensive bias testing that extends beyond simple demographic monitoring. Insist on transparent performance metrics and continuous oversight to prevent the deployment of unreliable or discriminatory systems. Your decisions should prioritize public trust and ethical safeguards over automated processes, ensuring models are not deployed without proven accuracy and fairness.

Key insights

Predictive policing models demand rigorous, continuous auditing and comprehensive bias testing for reliability and ethical deployment.

Principles

• Low precision scores indicate models flag many false positives.
• Shifting performance metrics suggest poor model governance.
• Monitoring ethnicity alone is insufficient for bias detection.

In practice

• Conduct independent audits of AI models.
• Implement continuous performance monitoring.
• Test for bias across multiple demographics.

Topics

• Predictive Policing
• AI Auditing
• Model Bias
• Performance Metrics
• Ethical AI
• Public Sector AI

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Policy Maker, Research Scientist

Related on AIssential

• How does live facial recognition work and how many UK police forces use it? — Live facial recognition is expanding in UK policing, raising efficiency versus civil liberties and bias concerns.
• AI Fairness Metrics Explained: A Practical Guide with Python — Measuring AI fairness requires selecting appropriate metrics as different definitions often conflict.
• Don’t trust Generative AI to do your taxes — and don’t trust it with people’s lives — AI systems are currently unreliable for critical governmental functions like tax processing and military applications.
• Digital monitoring is growing in South Africa’s public service – regulation needs to catch up — Rapid digital tool adoption in public sector evaluation outpaces ethical governance, creating risks for citizens and data integrity.
• AI incidents, audits, and the limits of benchmarks — AI safety requires systematic incident reporting and rigorous third-party evaluation to manage real-world risks.
• Carissa Véliz on the dangers of predictive AI — Predictive AI, designed for surveillance and prediction, risks social control and perpetuates historical biases, undermining democracy and fairness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by WIRED - Ai.