The Question Is the Contract

2025-07-29 · Source: Intentional Arrangement · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, extended

Summary

The article argues that modern system design largely ignores the fundamental role of questions, leading to information systems that fail to accurately answer user queries. Drawing on centuries of philosophical inquiry and library science principles, it highlights how traditional system design methodologies, focused on capabilities, are ill-suited for information retrieval systems like search engines, knowledge graphs, and AI agents. The author advocates for integrating "competency questions" (CQs), a rigorous methodology from ontology engineering, into general system design. CQs are natural-language questions with known correct answers, serving as both specifications and acceptance tests, ensuring that systems are purpose-built to address specific information needs and can be reliably evaluated for accuracy and completeness.

Key takeaway

For AI Product Managers designing knowledge-intensive systems, explicitly defining competency questions upfront is crucial. This discipline ensures your system's architecture, schema, and retrieval layers are purpose-built to answer specific user needs, preventing costly misfires and ensuring verifiable accuracy. Prioritize this conversation with domain experts and engineers to avoid implicit assumptions that lead to incomplete or incorrect results.

Key insights

Designing information systems around explicit questions, not just capabilities, ensures accurate and verifiable retrieval.

Principles

• Questions define the purpose of an information system.
• Competency questions serve as both specification and acceptance tests.
• The expressed question often differs from the real information need.

Method

Define natural-language competency questions with known answers before designing system architecture or schema. Use these CQs as acceptance tests and to define retrieval layer requirements, ensuring principled exclusion of unnecessary data.

In practice

• Define questions before schema design.
• Treat each question as an acceptance test.
• Use questions to define retrieval layer requirements.

Topics

• Competency Questions
• Information Retrieval
• Ontology Engineering
• System Design
• AI/ML Architectures

Best for: AI Product Manager, Software Engineer, AI Architect, Data Engineer

Related on AIssential

• When CQs Go Wrong: Challenges in CQ Verification with OE-Assist — Early refinement of Competency Questions (CQs) is crucial to prevent ambiguity and complexity in ontology engineering.
• The Right Problem — Solving the "right problem" too early, alone, and invisibly can precede significant market validation.
• The Semantic Layer and AI Agents // David Jayatillake // MLOps Podcast #343 — A semantic layer standardizes data definitions, enabling AI agents to automate data tasks and answer business questions efficiently.
• Ontology, Part II — Effective ontology design requires structured thinking, competency questions, design heuristics, and a three-box architecture.
• Not an Ontology — Major data platforms market "ontologies" that lack true reasoning and inference, failing to meet core definitional requirements.
• Designing Multi-Agent Deep Search Systems - 10 Seats Left — Deep search agents require multi-agent architectures for planning, execution, validation, merging, and iterative improvement.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Intentional Arrangement.