Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates - Archive ouverte HAL Access content directly
Conference Papers Year : 2022

Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates

Abstract

Automatic Audio Captioning (AAC) is the task that aims to describe an audio signal using natural language. AAC systems take as input an audio signal and output a free-form text sentence, called a caption. Evaluating such systems is not trivial, since there are many ways to express the same idea. For this reason, several complementary metrics, such as BLEU, CIDEr, SPICE and SPIDEr, are used to compare a single automatic caption to one or several captions of reference, produced by a human annotator. Nevertheless, an automatic system can produce several caption candidates, either using some randomness in the sentence generation process, or by considering the various competing hypothesized captions during decoding with beam-search, for instance. If we consider an end-user of an AAC system, presenting several captions instead of a single one seems relevant to provide some diversity, similarly to information retrieval systems. In this work, we explore the possibility to consider several predicted captions in the evaluation process instead of one. For this purpose, we propose SPIDEr-max, a metric that takes the maximum SPIDEr value among the scores of several caption candidates. To advocate for our metric, we report experiments on Clotho v2.1 and AudioCaps, with a transformed-based system. On AudioCaps for example, this system reached a SPIDEr-max value (with 5 candidates) close to the SPIDEr human score of reference.
Fichier principal
Vignette du fichier
IS_MY_AUTOMATIC_AUDIO_CAPTIONING_SYSTEM_SO_BAD__SPIDEr_max__A_METRIC_TO_CONSIDER_SEVERAL_CAPTION_CANDIDATES.pdf (285.72 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03810396 , version 1 (11-10-2022)
hal-03810396 , version 2 (14-11-2022)

Identifiers

Cite

Etienne Labbé, Thomas Pellegrini, Julien Pinquier. Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates. Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2022), Nov 2022, Nancy, France. ⟨hal-03810396v2⟩
68 View
28 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More