Hierarchical Fault Detection and Diagnosis for Transformer Architectures

2026-06-26 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

DEFault++, a hierarchical learning-based diagnostic technique, automates fault detection, categorization, and root-cause diagnosis for Transformer architectures. It identifies whether a fault is present, classifies it into one of 12 transformer-specific fault categories, and pinpoints the underlying root cause from up to 45 mechanisms. To facilitate training and evaluation, the researchers developed DEForm, a mutation technique, and constructed DEFault-bench, a benchmark of 3,739 labeled instances across seven transformer models and nine downstream tasks. DEFault++ measures runtime behavior at the component level, organizes data via a Fault Propagation Graph (FPG), and uses prototype matching with supervised contrastive learning. It achieves an AUROC over 0.96 for detection and a Macro-F1 over 0.85 for categorization and diagnosis. A developer study showed repair action accuracy increased from 57.1% to 83.3% with DEFault++ assistance.

Key takeaway

For Machine Learning Engineers debugging transformer models, DEFault++ provides a critical tool for identifying elusive, silent faults. Your teams should integrate this hierarchical diagnostic approach to move beyond generic DNN fault detection, pinpointing specific transformer component issues like QKV projection or masking faults. This can significantly improve repair action accuracy, as demonstrated by the 26.2% increase in developer study, reducing debugging time and improving model reliability.

Key insights

DEFault++ offers hierarchical, component-level fault diagnosis for Transformers using a Fault Propagation Graph.

Principles

• Transformer faults leave distinctive patterns within affected components.
• Inter-component dependencies can be modeled via a Fault Propagation Graph (FPG).
• Hierarchical diagnosis improves repair action accuracy for complex models.

Method

DEFault++ performs three-level diagnosis: fault detection, categorization into 12 types, and root-cause identification from 45 mechanisms. It uses component-level runtime measurements, structured by an FPG, and employs prototype matching with supervised contrastive learning.

In practice

• Use DEFault++ for diagnosing silent, transformer-specific faults.
• Leverage FPG-based explanations to trace fault propagation.
• Apply DEForm for systematic mutation testing of transformer models.

Topics

• Transformer Architectures
• Fault Detection
• Deep Learning Debugging
• Fault Diagnosis
• Mutation Testing
• Attention Mechanisms

Code references

• huggingface/diffusers
• SigmaJahan/DEFaultplusplus-Transformer-Debugging
• sigmaJahan/DEFaultPlusPlus

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale — A novel diagnostic identifies algebraic dead directions in LayerNorm transformers from parameters alone, without forward/backward passes.
• DDOR: Delta Debugging for Explainable Overrefusal Testing and Repair — DDOR uses delta debugging to explain and mitigate LLM overrefusal in black-box settings, improving usability and safety.
• The Ultimate Transformer Course for Working Engineers — The course offers practical understanding of transformers to debug LLM issues like slow inference and OOM errors.
• Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment — Lightweight Transformers can match traditional ML for on-device fault detection but demand significant resource tradeoffs, especially on imbalanced data.
• Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring — A layer-wise peeling framework diagnoses under-optimized transformer layers, revealing hidden inefficiencies during training.
• SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion — SpiralFormer uses multi-resolution recursion to learn hierarchical dependencies efficiently in looped Transformers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.