The Contribution of XAI for the Safe Development and Certification of AI: An Expert-Based Analysis

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Safety & Certification · Depth: Expert, extended

Summary

A study analyzed the potential and limitations of eXplainable AI (XAI) for safe AI development and certification, particularly in light of regulations like the EU AI Act. Researchers conducted 15 qualitative interviews with experts in XAI and certification from Germany, Austria, and Switzerland between January and March 2024. Findings indicate XAI methods are valuable for debugging and identifying biases or failures in ML models during development. However, experts largely view XAI's direct impact on formal AI certification as limited, citing challenges such as the difficulty in measuring "appropriate" explainability, the lack of robustness in current methods, and the "double black-box" issue where XAI explanations can also be opaque. The study highlights the need for clear certification requirements and user-centric XAI approaches.

Key takeaway

For Directors of AI/ML developing safety-critical systems, integrate eXplainable AI (XAI) primarily as a debugging and internal quality assurance tool to identify model biases and failures early. While XAI enhances development, do not rely on it as the sole solution for formal certification, as experts find its explainability difficult to measure and its methods often lack the robustness required for compliance. You should advocate for clear regulatory metrics and combine XAI with other verification methods for comprehensive AI safety.

Key insights

XAI aids AI development by revealing model flaws but faces significant limitations for formal certification due to measurement and robustness issues.

Principles

• AI certification requires comprehensive, correct information.
• XAI methods can identify ML model biases and failures.
• Explainability is not (and may not be) truly measurable.

Method

The study employed 15 qualitative, semi-structured interviews with experts in (X)AI and certification from Germany, Austria, or Switzerland, conducted via Microsoft Teams from January to March 2024, followed by inductive qualitative analysis.

In practice

• Use XAI for debugging ML systems during development.
• Conduct plausibility checks on models with XAI.
• Integrate XAI feedback into ML training processes.

Topics

• eXplainable AI
• AI Certification
• EU AI Act
• Machine Learning Safety
• Model Debugging
• Qualitative Interviews

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Legal Professional, Director of AI/ML

Related on AIssential

• People need to understand why systems make decisions, not merely whether the systems appear to perform well. A map of the Explainable AI (xAI) landscape. — Explainability is now a prerequisite for trustworthy AI, demanding tailored methods for diverse contexts.
• When Certificates Fail: A Unified Safety Framework for Embedded Neural Interface Models — Neural interface safety requires multi-objective auditing beyond single metrics, as certificates and task optimization can fail user welfare.
• Automated ML Explainability & Bias Testing in H2O.ai | Part 5 — Automated model explanation, bias analysis, and documentation are critical for responsible AI in regulated industries.
• What Should Be Done — Government's de facto AI licensing without clear standards stifles innovation and infrastructure, necessitating independent verification.
• Towards a Unified Multidimensional Explainability Metric: Evaluating Trustworthiness in AI Models — The framework unifies XAI evaluation by scoring fidelity, simplicity, and stability to build a knowledge base for transparent, trustworthy AI.
• AI incidents, audits, and the limits of benchmarks — AI safety requires systematic incident reporting and rigorous third-party evaluation to manage real-world risks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.