Performance measures for evolving predictions under delayed labelling classification - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Performance measures for evolving predictions under delayed labelling classification

Résumé

For many streaming classification tasks, the ground truth labels become available with a non-negligible latency. Given this delayed labelling setting, after the instance data arrives and before its true label is known, the online classifier model may change. Hence, the initial prediction can be replaced with additional periodic predictions gradually produced before the true label becomes available. The quality of these predictions may largely vary. Thus, the question arises of how to summarise the performance of these models when multiple predictions for a single instance are made due to delayed labels.In this study, we aim to provide intuitive performance measures summarising the performance of multiple predictions made for individual instances before their true labels arrive. Particular attention is paid to the fact that under the delayed label setting, the emphasis placed on the quality of initial predictions can vary depending on problem needs. The intermediate performance measures we propose complement existing initial and test-then-train performance evaluation when verification latency is observed. Results provided for both real and synthetic datasets show that the new measures can be used to easily rank methods in terms of their ability to produce and refine predictions before the true labels arrive.

Dates et versions

hal-04468444 , version 1 (20-02-2024)

Identifiants

Citer

Maciej Grzenda, Heitor Murilo Gomes, Albert Bifet. Performance measures for evolving predictions under delayed labelling classification. 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020, Jul 2020, Glasgow, United Kingdom. pp.1--8, ⟨10.1109/IJCNN48605.2020.9207256⟩. ⟨hal-04468444⟩
8 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More