Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

Leverage Large Language Models (LLMs) to automatically generate Competency Questions (CQs) for ontology engineering. This action pack helps automate the initial, human-intensive phase of defining requirements for knowledge base construction.

intermediate30 min6 steps

The play

Understand Competency Questions (CQs)
Familiarize yourself with Competency Questions. CQs are natural language questions that an ontology should be able to answer, used to define its scope and requirements. They bridge natural language needs with structured knowledge.
Define Your Ontology Domain and Scope
Clearly specify the domain (e.g., 'smart home devices', 'medical diagnoses') and the specific purpose or problem your ontology aims to address. This context is crucial for guiding the LLM.
Select an Appropriate LLM
Choose an LLM based on your needs. Consider factors like model size, cost, access (API vs. local), and whether you prefer open-source (e.g., Llama 3) or proprietary (e.g., GPT-4, Claude) models for your task.
Craft a Targeted CQ Generation Prompt
Write a clear and specific prompt for your chosen LLM. Instruct it to generate Competency Questions relevant to your defined domain. Include desired quantity, format, and any specific concepts to cover.
Generate and Critically Evaluate CQs
Run your prompt through the LLM. Carefully review the generated CQs for quality, relevance, consistency, and domain-specificity. Check if they effectively define the scope of your intended ontology.
Iterate and Refine Your Prompt
Based on your evaluation, refine your LLM prompt. Add more constraints, provide examples of good CQs, or specify aspects to improve. Rerun the generation process until the CQs meet your requirements.

Starter code

Generate 8-10 competency questions for an ontology describing 'supply chain logistics'. Focus on questions that address inventory management, transportation modes, and delivery timelines, and are suitable for a knowledge graph.

Source

Paperarxiv.org