AI as a Justice-System Risk Multiplier: (1) implementation mistakes, (2) questionable “fit” to the real-world population, and (3) legal/ethical fragility around data and proxies.

2025-11-28 · Source: Pascal’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, AI Ethics & Governance · Depth: Advanced, medium

Summary

In February 2026, the Dutch Justice and Security Inspectorate found that the probation services' OXREC algorithmic risk tool, used approximately 44,000 times annually, was operating irresponsibly. The Inspectorate identified software implementation errors dating back to 2018, specifically swapped formulas for detainees and non-detainees, and incorrect numbers, leading to systematically distorted (often too low) risk estimates, particularly for individuals with drug use issues and severe mental health conditions. Beyond these "formula mistakes," the tool relied on outdated data, was applied to a different population than intended, and failed to comply with privacy laws. Furthermore, OXREC utilized potentially discriminatory proxy variables like "neighborhood score" and "income level," which the Inspectorate had previously advised against without strict justification and safeguards. This combination of errors, questionable fit, and ethical fragility created systemic distortions in the justice system.

Key takeaway

For CTOs and VPs of Engineering overseeing critical algorithmic deployments, this case highlights that even minor technical errors can become systemic risks when scaled. You must implement robust governance, continuous monitoring, and strict validation protocols for any algorithm impacting liberty or public safety. Ensure your procurement contracts mandate auditability and liability, and that your teams are equipped for incident response, treating these systems with the same rigor as critical infrastructure to prevent widespread harm and institutional mistrust.

Key insights

Small algorithmic errors in justice systems scale into systemic distortions with severe societal and individual consequences.

Principles

• Algorithmic tools require safety-style regulation.
• Continuous lifecycle supervision is crucial for high-impact algorithms.
• "Human in the loop" must be a measurable control.

Method

Regulators should mandate formal software assurance, incident response protocols, continuous monitoring, and periodic re-certification for justice risk tools, alongside strict rules for data and proxy variables.

In practice

• Implement automated testing and regression tests.
• Prohibit tool use outside its validated population.
• Require necessity justification for proxy variables.

Topics

• Recidivism Risk Algorithms
• Algorithmic Bias
• AI Governance
• Criminal Justice AI
• AI System Safety

Best for: CTO, VP of Engineering/Data, Executive, AI Ethicist, Policy Maker, Legal Professional

Related on AIssential

• 'Silent failure at scale': The AI risk that can tip the business world into disorder - CNBC — AI's greatest risk stems from silent, compounding errors due to complexity exceeding human comprehension, not rogue agents.
• Meta. When a platform’s commercial success depends on measuring, steering, and compounding human attention, it eventually drifts toward the most reliable drivers of attention: — Optimizing for engagement at scale without strong accountability creates structural conflicts of interest and predictable harms.
• AI Warfare Is Outpacing Our Ability to Control It — AI in warfare outpaces governance, leading to civilian harm, automation bias, and ethical dilemmas.
• LexisNexis Webinar: AI Risk, Governance + Adoption — Responsible AI adoption requires balancing innovation with robust governance and risk management, particularly for CTOs in legal tech.
• The Next Evolution of Fairness: Why LTAE and LAAM are expanding into AI Safety. — AI safety's structural failures, mirroring human oversight issues, necessitate governance frameworks like SLAAM to prevent asymmetric enforcement and anchor drift.
• Human layer of AI: How to build human-centered AI safety to mitigate harm and misuse — Effective AI safety requires proactive risk mapping and empowered safety processes to prevent harm and misuse.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Pascal’s Substack.