Results

Abstract

The availability of Large Language Models (LLMs) presents a unique opportunity to reinvigorate research on Knowledge Engineering (KE) automation. This trend is already evident in recent efforts developing LLM-based methods and tools for the automatic generation of Competency Questions (CQs), natural language questions used by ontology engineers to define the functional requirements of an ontology. However, the evaluation of these tools lacks standardisation. This undermines methodological rigor and hinders the replication and comparison of results. To address this gap, we introduce Bench4KE, an extensible API-based benchmarking system for KE automation. The presented release focuses on evaluating tools that generate CQs automatically. Bench4KE provides a curated gold standard consisting of CQ datasets from 17 real-world ontology engineering projects and uses a suite of similarity metrics to assess the quality of the CQs generated. We present a comparative analysis of 6 recent CQ generation systems, which are based on LLMs, establishing a baseline for future research. Bench4KE is also designed to accommodate additional KE automation tasks, such as SPARQL query generation, ontology testing and drafting. Code and datasets are publicly available under the Apache 2.0 license.

Abstract

This paper introduces the Cultural Heritage Survey Ontology Design Pattern (CHS-ODP), a content pattern that abstracts from specific implementations and provides a reference core for modelling survey processes within cultural heritage knowledge graphs. The pattern is grounded in a set of generalised competency questions derived from two independent real-world use cases: a Reflectance Transformation Imaging (RTI)-based documentation campaign on rock art and a diagnostic investigation on illuminated manuscripts. Designed in alignment with CIDOC-CRM, ArCo, DOLCE UltraLight, the proposed pattern supports interoperable and reusable survey-centric modelling across heterogeneous cultural-heritage domains.