Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

2026-04-17 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

A new study characterizes Competency Questions (CQs) generated by various Large Language Models (LLMs) for ontology engineering. Traditionally, CQs are manually created by ontology engineers and domain experts to define requirements as natural language questions an ontology must satisfy. This research introduces quantitative measures to systematically compare LLM-generated CQs across multiple dimensions, including readability, relevance to input text, and structural complexity. The study evaluates both open models like KimiK2-1T, LLama3.1-8B, and LLama3.2-3B, and closed models such as Gemini 2.5 Pro and GPT 4.1. Experiments conducted across diverse use cases and requirements reveal that LLM performance in CQ generation produces distinct profiles influenced by the specific use case, highlighting the varied intrinsic properties of the CQs produced.

Key takeaway

For ontology engineers and AI scientists evaluating LLMs for automated Competency Question generation, you should conduct a systematic, cross-domain analysis of generated CQs. Focus on readability, relevance, and structural complexity, as model performance and output characteristics are highly dependent on the specific use case. This will help you select the most appropriate LLM for your project's requirements.

Key insights

LLMs can automate Competency Question generation, but their output quality varies significantly by model and use case.

Principles

• CQ generation profiles are use case-dependent.
• LLMs democratize ontology engineering access.

Method

The study uses quantitative measures to compare LLM-generated CQs based on readability, relevance to input, and structural complexity across open and closed models for various use cases.

In practice

• Evaluate LLM-generated CQs for specific use cases.
• Consider both open and closed LLMs for CQ generation.

Topics

• Competency Questions
• Ontology Engineering
• Large Language Models
• CQ Generation Automation
• LLM Performance Analysis

Best for: AI Scientist, Research Scientist, NLP Engineer

Related on AIssential

• Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models — CompCQ framework systematically characterizes LLM-generated Competency Questions across multiple linguistic, structural, and semantic dimensions.
• SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity — LLMs moderately generate equations from text but struggle with semantic accuracy and reliable self-evaluation.
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions — Multi-Legal-Bench reveals LLM legal reasoning varies significantly across jurisdictions and tasks, with transfer quality tied to label-set alignment.
• When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering — LLMs require retrieval augmentation and strong reasoning for reliable off-guideline medical question answering.
• Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware — Code-tuned general-purpose LLMs outperform domain-specific medical LLMs for T-SQL generation on manufacturing schemas.
• Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language — Automating executable visual workflow generation from natural language remains a significant challenge for current LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.