Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

2026-07-07 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, quick

Summary

A new black-box evaluation framework systematically assesses Large Language Models' (LLMs) capability to generate Design Structure Matrices (DSMs) from structured technical documentation. This framework benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs) using a reproducible methodology. It integrates structural metrics (Completeness, Correctness, Coupling Density), classification metrics (Selective Accuracy, Abstention Coverage), and stability measures (Entropy, Fleiss' κ), synthesizing them into a Composite Quality Score (Q). Controlled experiments on a fictive abstract system and a real-world refrigerator decomposition revealed that LLMs can produce structurally plausible DSMs with high reproducibility under well-structured inputs. However, LLMs remain sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation, leading to systematic hallucination and abstention failures. The framework offers a transparent benchmark for auditing Auto-DSM pipelines and lays groundwork for integrating LLM-based decomposition into model-based systems engineering (MBSE) workflows.

Key takeaway

For AI Engineers evaluating LLM-based system decomposition, you should rigorously benchmark Auto-DSM pipelines using a comprehensive framework like the one proposed. Recognize that while LLMs can generate plausible DSMs from well-structured inputs, their sensitivity to ambiguity and prompt variations necessitates careful input curation. Prioritize clear dependency definitions and structured documentation to mitigate hallucination and abstention failures, ensuring reliable integration into MBSE workflows.

Key insights

A black-box framework evaluates LLM-generated Design Structure Matrices (DSMs), revealing potential and limitations in automation for model-based systems engineering.

Principles

• LLMs generate plausible DSMs from structured inputs.
• Ambiguity and prompt formulation impact LLM DSM accuracy.
• Reproducibility is high with well-defined inputs.

Method

The framework benchmarks LLM-generated DSMs against ground-truth matrices using structural, classification, and stability metrics, synthesized into a Composite Quality Score (Q).

In practice

• Audit Auto-DSM pipelines with the proposed framework.
• Integrate LLM-based decomposition into MBSE workflows.
• Identify sources of LLM hallucination in DSM generation.

Topics

• LLM Evaluation
• Design Structure Matrices
• Model-Based Systems Engineering
• Auto-DSM
• System Decomposition
• LLM Hallucination

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Retrieval Augmented (Knowledge Graph), and Large Language Model-Driven Design Structure Matrix (DSM) Generation of Cyber-Physical Systems — LLMs, especially with RAG and GraphRAG, can automate Design Structure Matrix generation for complex systems.
• # I Got Tired of Saying “This Response Is Better” Without Being Able to Prove It — Combining human domain expertise with automated pattern detection improves LLM evaluation accuracy and provides objective vocabulary for subjective judgments.
• Moving beyond manual prompting: A practical introduction to DSPy — DSPy offers a programmatic, declarative framework for building robust, optimizable LLM applications, moving beyond manual prompt engineering.
• A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models — Black-box uncertainty estimation for LLMs lacks a dominant method, but candidate comparison and hybrid approaches show promise.
• A Framework for Evaluating Agentic Skills at Scale — A new framework rigorously evaluates LLM agent skills at scale, revealing varied model adherence and significant behavioral changes.
• Dimension-Level Intent Fidelity Evaluation for Large Language Models: Evidence from Structured Prompt Ablation — Dimension-level intent fidelity evaluation uncovers LLM quality deficits missed by holistic assessment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.