Can LLMs Learn and Apply Multi-Level Modelling Semantics? A First Empirical Study

2026-07-16 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

This study presents the first empirical investigation into Large Language Models' (LLMs) ability to learn and apply Multi-Level Modelling (MLM) semantics, a complex domain crucial for Industry 5.0's human-centric system design. Researchers evaluated three commercial LLMs—GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro—across six prompting strategies. They generated 90 multi-level models for the MULTI Warehouse Challenge in the SLICER language, providing full semantic documentation in context. While 89 of 90 models achieved syntactic correctness, semantic correctness was only partially met, with Instantiation/Specialisation Correctness ranging from 52% to 79%. LLMs reliably reproduced explicitly stated content but struggled to infer implied structure or constraints. Prompting strategies showed trade-offs between precision and completeness, and self-checking primarily functioned as a rule checker. Claude Opus 4.6 demonstrated the most balanced performance.

Key takeaway

For AI Engineers designing human-centric modelling workflows, this study highlights that while LLMs can generate syntactically correct multi-level models and core structures, their semantic reliability for fine-grained relation choices remains partial. You should delegate structural generation to LLMs but implement robust human expert review or automated axiom checking for semantic correctness. Do not assume LLMs will infer implicit constraints or complete unstated structural details, as this is a consistent failure pattern.

Key insights

LLMs reliably transcribe explicit modelling requirements but struggle with implicit multi-level semantics and inferring unstated structural safeguards.

Principles

• LLMs transcribe explicit content better than inferring implicit structure.
• Relation-kind selection is a key bottleneck for LLM-assisted modelling.
• Multi-level modelling introduces unique relation-kind error types.

Method

A 5-phase experimental design evaluated LLMs on SLICER model generation, including input preparation, LLM selection, 6 prompting strategies, 5 runs per config, and 14-metric evaluation against a reference.

In practice

• Favor zero-shot or few-shot prompting for human-expert-completed drafts.
• Use step-by-step guidance for rule-compliant output for automated processing.
• Integrate automated axiom checkers for precise, rule-level feedback.

Topics

• Large Language Models
• Multi-Level Modelling
• SLICER Language
• Prompting Strategies
• Model Generation
• Semantic Correctness

Best for: AI Scientist, Research Scientist, AI Engineer

Related on AIssential

• SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity — LLMs moderately generate equations from text but struggle with semantic accuracy and reliable self-evaluation.
• How Large Language Models Actually Work (Explained Simply) — LLMs are statistical prediction engines, not reasoning entities, whose capabilities emerge from massive scale and sophisticated attention mechanisms.
• Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures — LLMs' apparent faithfulness to intermediate reasoning structures is fragile; they often fail to update predictions after causal intervention.
• A Roadmap for Tamed Interactions with Large Language Models — Software Engineering principles can formalize LLM interactions, ensuring reliable and structured outputs through a dedicated scripting language.
• Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation — LLMs achieve only 8.6% semantic correctness for TLA+ specifications, primarily with progressive prompting.
• Agentic Large Language Models for Automated Structural Analysis of 3D Frame Systems — Agentic LLMs with structured decomposition and specialized agents can reliably automate complex 3D structural analysis tasks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.