When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

2026-06-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Knowledge Representation & Ontologies · Depth: Expert, quick

Summary

Competency Questions (CQs) are fundamental to CQ-verification, a process evaluating ontologies against natural language questions to confirm proper modeling. This verification is often time-consuming and error-prone due to complex linguistic interpretation, precise alignment with formal ontology constructs, and inherent ambiguities within CQs. Researchers investigated factors making CQs challenging and explored solutions to improve user performance in CQ-verification. An experiment involved 19 participants conducting CQ-verification across 20 tasks, utilizing an LLM assistant, OE-Assist, to support ontology evaluation. The study's findings highlight the critical need for a dedicated tool to refine CQs before their publication, thereby mitigating ambiguity and excessive complexity in subsequent stages of the ontology engineering process.

Key takeaway

For ontology engineers and AI scientists involved in designing or evaluating ontologies, this research underscores the critical importance of refining Competency Questions (CQs) early. Ambiguous or complex CQs lead to time-consuming and error-prone verification processes. You should integrate dedicated tools or rigorous review steps to clarify CQs before their formal publication, preventing downstream issues and ensuring more consistent ontology modeling and evaluation outcomes.

Key insights

Early refinement of Competency Questions (CQs) is crucial to prevent ambiguity and complexity in ontology engineering.

Principles

• CQ-verification is inherently complex and error-prone.
• Linguistic nuances significantly impact ontology evaluation.
• Pre-publication CQ refinement enhances process reliability.

Method

Participants performed CQ-verification on 20 tasks, supported by an LLM assistant (OE-Assist) for ontology evaluation, to identify CQ challenges.

In practice

• Implement a tool for CQ refinement before publishing.
• Focus on clarifying linguistic ambiguities in CQs.

Topics

• Competency Questions
• Ontology Engineering
• CQ Verification
• LLM Assistants
• Ontology Evaluation
• Knowledge Representation

Best for: AI Scientist, Research Scientist

Related on AIssential

• Ontology, Part II — Effective ontology design requires structured thinking, competency questions, design heuristics, and a three-box architecture.
• The Question Is the Contract — Designing information systems around explicit questions, not just capabilities, ensures accurate and verifiable retrieval.
• Abstractions of Queries in Ontology-Based Data Access — The paper extends UCQs to express complete query abstractions in OBDA without increasing complexity.
• The Open World/Closed World Conundrum — The semantic web is shifting from Open World to Closed World Assumptions, driven by practical system demands.
• Semantic Foundations for Reliable Enterprise AI — Semantic clarity in data, achieved through robust data contracts and ontologies, is crucial for reliable enterprise AI outputs.
• Why ontologies matter, why they fail and how to build them anyway — Treating ontologies as products, scoped to use cases and continuously maintained, is crucial for successful enterprise AI semantic understanding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.