Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Expert, quick

Summary

A new black-box evaluation framework systematically assesses Large Language Models' (LLMs) capability to generate Design Structure Matrices (DSMs) from structured technical documentation. Addressing the closed-source nature of existing Auto-DSM pipelines, this framework benchmarks LLM-generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs). It integrates single-run and multi-run evaluations using structural metrics (Completeness, Correctness, Coupling Density), classification metrics (Selective Accuracy, Abstention Coverage), and stability measures (Entropy, Fleiss' $κ$), culminating in a Composite Quality Score (Q). Experiments on a fictive abstract system and a real-world refrigerator decomposition dataset reveal LLMs can produce structurally plausible DSMs with high reproducibility under well-structured inputs. However, they remain sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation, leading to systematic hallucination and abstention failures. The framework offers a transparent benchmark for auditing Auto-DSM pipelines and supports integrating LLM-based decomposition into model-based systems engineering (MBSE) workflows.

Key takeaway

For Machine Learning Engineers developing Auto-DSM pipelines or integrating LLMs into MBSE workflows, you should recognize LLMs' potential for generating structurally plausible DSMs from well-structured inputs. However, be vigilant about input ambiguity and prompt sensitivity, which can lead to hallucination. Utilize the proposed black-box evaluation framework, incorporating structural, classification, and stability metrics, to rigorously audit your LLM-based decomposition methods and ensure reliable, high-quality outputs.

Key insights

LLMs show promise for DSM generation but are sensitive to input quality and prompt design, necessitating systematic black-box evaluation.

Principles

• LLMs generate plausible DSMs from structured inputs.
• Input ambiguity and prompt design affect DSM quality.
• Evaluate DSMs using structural, classification, and stability metrics.

Method

Benchmark LLM-generated DSMs against ground-truth matrices using structural, classification, and stability metrics. Synthesize results into a Composite Quality Score (Q) for comprehensive black-box evaluation.

In practice

• Audit Auto-DSM pipelines using this framework.
• Integrate LLM decomposition into MBSE workflows.
• Prioritize clear, unambiguous input for LLM DSMs.

Topics

• Large Language Models
• Design Structure Matrix
• Black-Box Evaluation
• Model-Based Systems Engineering
• Prompt Engineering
• LLM Hallucination

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• LLMs just have more time and energy? — LLMs imitate human language patterns, including confidence and humility, through relentless, large-scale data compression, acting as a mirror of human communication.
• Retrieval Augmented (Knowledge Graph), and Large Language Model-Driven Design Structure Matrix (DSM) Generation of Cyber-Physical Systems — LLMs, especially with RAG and GraphRAG, can automate Design Structure Matrix generation for complex systems.
• LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models — Linguistic tone systematically influences VLM hallucination, with varying effects on frequency and intensity across models and tasks.
• Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement — Decomposing LLM evaluation into binary questions provides interpretable, multi-dimensional scores and enables self-improvement.
• SWE-IF: Aligning Code Evaluation with Human Preference — Human preference in code generation requires evaluating both functional correctness and instruction following, not just functionality.
• Dimension-Level Intent Fidelity Evaluation for Large Language Models: Evidence from Structured Prompt Ablation — Dimension-level intent fidelity evaluation uncovers LLM quality deficits missed by holistic assessment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.