Ontology matching benchmarks: generation, stability, and discriminability - Archive ouverte HAL Access content directly
Journal Articles Journal of Web Semantics Year : 2013

Ontology matching benchmarks: generation, stability, and discriminability


The OAEI Benchmark test set has been used for many years as a main reference to evaluate and compare ontology matching systems. However, this test set has barely varied since 2004 and has become a relatively easy task for matchers. In this paper, we present the design of a flexible test generator based on an extensible set of alterators which may be used programmatically for generating different test sets from different seed ontologies and different alteration modalities. It has been used for reproducing Benchmark both with the original seed ontology and with other ontologies. This highlights the remarkable stability of results over different generations and the preservation of difficulty across seed ontologies, as well as a systematic bias towards the initial Benchmark test set and the inability of such tests to identify an overall winning matcher. These were exactly the properties for which Benchmark had been designed. Furthermore, the generator has been used for providing new test sets aiming at increasing the difficulty and discriminability of Benchmark. Although difficulty may be easily increased with the generator, attempts to increase discriminability proved unfruitful. However, efforts towards this goal raise questions about the very nature of discriminability.
Fichier principal
Vignette du fichier
euzenat2013a.pdf (338.52 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-01123792 , version 2 (12-12-2013)
hal-01123792 , version 1 (05-03-2015)



Jérôme Euzenat, Maria Rosoiu, Cassia Trojahn dos Santos. Ontology matching benchmarks: generation, stability, and discriminability. Journal of Web Semantics, 2013, 21, pp.30-48. ⟨10.1016/j.websem.2013.05.002⟩. ⟨hal-01123792v2⟩
466 View
838 Download



Gmail Facebook X LinkedIn More