From Failure to Alignment: A Requirements Engineering Framework for Machine Learning Systems

2026-07-01 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

The REAL (Requirements Engineering for mAchines that Learn - and Fail) framework is proposed for developing trustworthy Machine Learning Systems (MLS) that align with stakeholder needs. This model-based framework integrates requirements for data, models, and the system, uses failure to drive the exploration of alternative requirements, and employs iterative, traceable refinement. Addressing challenges like learned behavior and inherent failure, REAL operationalizes requirement satisfaction by linking stakeholder goals to executable domain instantiations, simulation traces, and observed violations. It involves identifying failures via grammar-guided scenario exploration, analyzing them as obstacles using KAOS-style goal models, and mitigating them through multi-layer adaptation across data, model, system, and requirement levels. Demonstrated on an autonomous braking system using CARLA, Scenic, and YOLOv5, REAL systematically reduced child-collision rates from 100% to 8% and eliminated nominal-weather failures, highlighting the necessity of coordinated cross-layer adaptation.

Key takeaway

For Machine Learning Engineers developing safety-critical systems, you should integrate a failure-driven requirements engineering framework like REAL. This approach systematically links scenario-based testing outcomes to requirement refinement, enabling you to identify and analyze misalignments between stakeholder goals, domain assumptions, and system behavior. By applying multi-layer adaptations across data, model, system, and requirements, you can achieve more robust alignment and significantly reduce critical failure rates, as demonstrated by the 92% reduction in child-collision rates.

Key insights

REAL systematically integrates failure analysis into requirements engineering for ML systems, ensuring alignment with stakeholder needs.

Principles

• Weave data, model, and system requirements together.
• Use failure to drive exploration of alternative requirements.
• Iteratively refine MLS requirements with traceability.

Method

REAL identifies failures via grammar-guided scenario exploration, analyzes them as obstacles in a KAOS-style goal model, and mitigates through multi-layer adaptation (data, model, system, requirements) within an iterative alignment loop.

In practice

• Employ grammar-guided scenario generation for systematic failure discovery.
• Interpret empirical failures as obstacles in goal models for diagnosis.
• Apply multi-layer adaptation (data, model, system, requirements) for mitigation.

Topics

• Requirements Engineering
• Machine Learning Systems
• Failure Analysis
• Autonomous Driving
• Goal Modeling
• Multi-layer Adaptation

Code references

• darkaengl/REAL
• ApolloAuto/apollo
• microsoft/responsible-ai-toolbox

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• harvard-edge / cs249r_book — AI engineering focuses on building robust, real-world intelligent systems, not just isolated models.
• Using Data Contracts as a Value Assessment Framework for Data or AI Initiatives — Data contracts are enforceable promises with consequences, crucial for data product accountability and AI system reliability.
• A Taxonomy of Runtime Faults in Model Context Protocol Servers — A new taxonomy empirically categorizes 73 runtime fault types in Model Context Protocol (MCP) servers, enhancing reliability understanding for LLM tool integration.
• AI Code Review Only Catches Half of Your Bugs — AI code review needs explicit requirements and intent to catch design flaws and security vulnerabilities.
• Why Most AI Models Fail Before Training Even Begins — Most AI failures originate from data and pipeline issues, not model algorithms, requiring an operational discipline.
• The Trust Layer: How Great Engineering Teams Make AI Systems Reliable — LLMs fail silently by being confidently wrong, demanding a new engineering posture focused on continuous correctness monitoring.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.