A Benchmark for Fact Checking Algorithms Built on Knowledge Bases - Archive ouverte HAL Access content directly
Conference Papers Year : 2019

A Benchmark for Fact Checking Algorithms Built on Knowledge Bases

Paolo Papotti


Fact checking is the task of determining if a given claim holds. Several algorithms have been developed to check claims with reference information in the form of facts in a knowledge base. While individual algorithms have been experimentally evaluated in the past, we provide the first comprehensive and publicly available benchmark infrastructure for evaluating methods across a wide range of assumptions about the claims and the reference information. We show how, by changing the popularity, transparency, homogeneity, and functionality properties of the facts in an experiment, it is possible to influence significantly the performance of the fact checking algorithms. We introduce a benchmark framework to systematically enforce such properties in training and testing datasets with fine tune control over their properties. We then use our benchmark to compare fact checking algorithms with one another, as well as with methods that can solve the link prediction task in knowledge bases. Our evaluation shows the impact of the four data properties on the qualitative performance of the fact checking solutions and reveals a number of new insights concerning their applicability and performance.
No file

Dates and versions

hal-03560733 , version 1 (07-02-2022)



Viet-Phi Huynh, Paolo Papotti. A Benchmark for Fact Checking Algorithms Built on Knowledge Bases. CIKM 2019, 28th ACM International Conference on Information and Knowledge Management, Nov 2019, Beijing, China. pp.689-698, ⟨10.1145/3357384.3358036⟩. ⟨hal-03560733⟩


12 View
0 Download



Gmail Facebook X LinkedIn More