Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation

2026-03-18 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

R2ABench, a novel benchmark, has been introduced to evaluate Large Language Models' (LLMs) capability in generating software architecture diagrams directly from unstructured Product Requirements Documents (PRDs). Comprising 17 real-world software projects (9 Python, 8 Java) with comprehensive PRDs and expert-curated PlantUML reference diagrams, R2ABench also features a multi-dimensional, hybrid evaluation framework. This framework assesses generated diagrams across Structural Graph Metrics, Multi-dimensional Scoring, and Architecture Anti-pattern Detection. An empirical study using this benchmark revealed that LLMs achieve strong syntactic validity and entity extraction (Node F1 around 0.5) but fundamentally struggle with relational reasoning (Edge F1 below 0.2), resulting in fragmented architectures. Code-specialized models like Qwen3-coder-480b (Edge F1 0.1760) partially mitigate this, while agent frameworks often introduce instability rather than consistent improvements.

Key takeaway

For AI Architects evaluating LLM-driven software architecture generation, recognize that current models excel at component identification but severely underperform in establishing correct inter-component relationships. While code-specialized models offer slight improvements, agentic frameworks introduce significant instability. You should prioritize solutions that explicitly address relational modeling failures and topological coherence, rather than relying solely on entity extraction or general-purpose agents, to avoid fragmented and unmaintainable designs.

Key insights

LLMs excel at extracting architectural entities but fail at relational modeling, leading to fragmented software designs.

Principles

• LLMs over-abstract PRD functions to architectural nodes.
• LLMs struggle to infer dynamic behaviors from static requirements.
• Task ambiguity causes LLMs to conflate business logic with implementation.

Method

The R2ABench methodology involves dataset construction from real-world projects, context gradation testing (Full, -Arch, Min), and a multi-dimensional evaluation framework.

In practice

• Use R2ABench to benchmark LLM architecture generation.
• Employ hybrid evaluation for structural and semantic correctness.
• Detect anti-patterns like isolated or "God" components.

Topics

• Large Language Models
• Software Architecture
• Architectural Benchmarking
• PlantUML Diagrams
• Software Requirements
• Architectural Anti-patterns

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models — LLM architectural knowledge is high overall but uneven, with context length effects varying by task type.
• BugBash'26 Afternoon of Day 1 — LLMs make code cheap, but core software design, correctness, and reliable system engineering remain human-centric challenges.
• Governed Shared Memory for Multi-Agent LLM Systems — Governed shared memory for multi-agent LLM systems requires explicit systems-level abstractions to prevent critical failure modes.
• Panel: Large Language Models — LLMs are powerful prototyping tools, but enterprise production demands rigorous data quality, security, and integration beyond current capabilities.
• SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems — Generative AI struggles to formally model complex concurrent systems, particularly with temporal reasoning and abstraction.
• Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment — Agentic LLM judges offer a scalable solution for evaluating and enhancing architectural reasoning in code generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.