Difficulty and Severity-Oriented Metrics for Test Prioritization in Deep Learning Systems - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Difficulty and Severity-Oriented Metrics for Test Prioritization in Deep Learning Systems

Hamzah Al-Qadasi
  • Fonction : Auteur
  • PersonId : 1280505
Saddek Bensalem
  • Fonction : Auteur

Résumé

Recently, there has been a growing trend in AI testing toward developing new test prioritization algorithms for deep learning systems. These algorithms aim to reduce the cost and time needed to annotate test datasets by prioritizing instances with a higher chance of exposing faults. Various metrics have been used to evaluate the effectiveness of these algorithms, e.g., APFD, RAUC, and ATRC. However, there is a lack of research to confirm their validity. The results indicate that the existing metrics have severe limitations. For example, some metrics ignore the labeling budget and prioritize the fault detection rate instead of the fault detection ratio. Moreover, others overlook the prioritization difficulty in the evaluation. As a solution, we develop a new metric (WFDR), which solves the deficiencies of previous metrics. We also draw attention to a new research area, known as severity prioritization, which emphasizes the importance of prioritizing misclassified instances according to the severity level, particularly in critical situations. Our experiments reveal that instances with high severity make up more than 20% of all misclassified instances. Thus, these instances should be prioritized when it comes to labeling. Consequently, we proposed a new metric known as (SFDR) that evaluates the effectiveness of algorithms in prioritizing high-severity instances. Our evaluations show that our proposed metrics are more effective than other existing metrics. Besides, our two metrics re-evaluate some recent algorithms and indicate that these algorithms perform poorly.
Fichier principal
Vignette du fichier
IEEE_AI_TEST_2023 (1).pdf (2.21 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04196591 , version 1 (05-09-2023)

Identifiants

Citer

Hamzah Al-Qadasi, Yliès Falcone, Saddek Bensalem. Difficulty and Severity-Oriented Metrics for Test Prioritization in Deep Learning Systems. 5th IEEE International Conference on Artificial Intelligence (AITest 2023), Jul 2023, Athènes, Greece. pp.40-48, ⟨10.1109/AITest58265.2023.00015⟩. ⟨hal-04196591⟩
14 Consultations
47 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More