Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates

Résumé

Automatic Audio Captioning (AAC) is the task that aims to describe an audio signal using natural language. AAC systems take as input an audio signal and output a free-form text sentence, called a caption. Evaluating such systems is not trivial, since there are many ways to express the same idea. For this reason, several complementary metrics, such as BLEU, CIDEr, SPICE and SPIDEr, are used to compare a single automatic caption to one or several captions of reference, produced by a human annotator. Nevertheless, an automatic system can produce several caption candidates, either using some randomness in the sentence generation process, or by considering the various competing hypothesized captions during decoding with beam-search, for instance. If we consider an end-user of an AAC system, presenting several captions instead of a single one seems relevant to provide some diversity, similarly to information retrieval systems. In this work, we explore the possibility to consider several predicted captions in the evaluation process instead of one. For this purpose, we propose SPIDEr-max, a metric that takes the maximum SPIDEr value among the scores of several caption candidates. To advocate for our metric, we report experiments on Clotho v2.1 and AudioCaps, with a transformed-based system. On AudioCaps for example, this system reached a SPIDEr-max value (with 5 candidates) close to the SPIDEr human score of reference.
Fichier principal
Vignette du fichier
IS_MY_AUTOMATIC_AUDIO_CAPTIONING_SYSTEM_SO_BAD__SPIDEr_max__A_METRIC_TO_CONSIDER_SEVERAL_CAPTION_CANDIDATES.pdf (285.72 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03810396 , version 1 (11-10-2022)
hal-03810396 , version 2 (14-11-2022)

Identifiants

Citer

Etienne Labbé, Thomas Pellegrini, Julien Pinquier. Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates. Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2022), Nov 2022, Nancy, France. ⟨hal-03810396v2⟩
120 Consultations
53 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More