Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements

2026-05-21 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Construction Technology & Building · Depth: Advanced, extended

Summary

Ishigaki-IDS-Bench is a new benchmark designed to evaluate Large Language Models' (LLMs) ability to generate Information Delivery Specification (IDS) XML from Building Information Modeling (BIM) information requirements. This benchmark addresses the challenge of LLMs producing structured outputs that must adhere to industry-standard XML, domain vocabularies like Industry Foundation Classes (IFC), and external validation tools. It comprises 166 expert-authored and verified examples, derived from 83 practical scenarios in both Japanese and English, each with a corresponding gold IDS file and metadata. The evaluation protocol involves a two-stage process: IDSAuditTool-based audits for formal validity (Processability, Structure, Content) and facet-level content agreement against gold IDS. Zero-shot evaluation across 10 LLMs revealed that the best model, GPT-5.5, achieved 65.6% macro F1 for content agreement, yet only 27.7% of outputs passed the Content audit. These results indicate LLMs can partially capture information requirements but struggle with stable, standard-compliant XML generation. The benchmark and evaluation scripts are publicly available under a CC BY 4.0 license.

Key takeaway

For Machine Learning Engineers building LLM-based structured generation systems, recognize that current models, including GPT-5.5, struggle with industry-standard XML and domain vocabulary constraints. You should prioritize developing robust methods for standard conformance, especially for property-set names and value constraints. Implement interactive workflows or utilize structured inputs like CSV to enhance output reliability and minimize formal errors.

Key insights

LLMs struggle with domain-standard structured generation requiring simultaneous syntax, vocabulary, and external validation.

Principles

• Domain-specific XML generation needs multi-layered constraint satisfaction.
• Formal validity and content agreement are distinct evaluation metrics.
• Interactive updates improve IDS generation over generation from scratch.

Method

A two-stage evaluation protocol combines IDSAuditTool audits for formal validity (Processability, Structure, Content) with facet-level content agreement against gold IDS files.

In practice

• Use multi-turn settings for IDS generation tasks.
• Prefer CSV inputs over natural language for requirements.
• Focus on property-set names, cardinality, value constraints.

Topics

• Large Language Models
• Structured Generation
• Building Information Modeling
• Information Delivery Specification
• IFC Vocabulary
• Benchmark Evaluation

Code references

• onestruction/Ishigaki-IDS-Bench
• buildingSMART/IDS-Audit-tool

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware — Code-tuned general-purpose LLMs outperform domain-specific medical LLMs for T-SQL generation on manufacturing schemas.
• ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction — LLMs struggle with complex, enterprise-scale PDF-to-JSON extraction, especially as schema breadth increases.
• DALM: A Domain-Algebraic Language Model via Three-Phase Structured Generation — DALM uses domain algebra and structured denoising to prevent cross-domain contamination and hallucination in language models.
• aishwaryanr / awesome-generative-ai-guide — This repository centralizes diverse generative AI learning and development resources.
• Beyond Vector Similarity: A Structural Analysis of Graph-Augmented Retrieval for Industrial Knowledge Graphs — Effective LLM-based graph reasoning hinges on providing robust computational operators, not just enhanced model intelligence.
• Structured Outputs with LLMs: JSON Mode, Function Calling, and When to Use Each — LLM structured output methods vary in schema enforcement, suiting different application reliability and consistency needs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.