Killing two birds with one stone: can an audio captioning system also be used for audio-test retrieval? - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Killing two birds with one stone: can an audio captioning system also be used for audio-test retrieval?

Faire d'une pierre deux coups : un système de description de l'audio peut-il être utilisé pour la recherche audio-textuelle ?

Résumé

Automated Audio Captioning (AAC) aims to develop systems capable of describing an audio recording using a textual sentence. In contrast, Audio-Text Retrieval (ATR) systems seek to find the best matching audio recording(s) for a given textual query (Text-to-Audio) or vice versa (Audio-to-Text). These tasks require different types of systems: AAC employs a sequence-to-sequence model, while ATR utilizes a ranking model that compares audio and text representations within a shared projection subspace. However, this work investigates the relationship between AAC and ATR by exploring the ATR capabilities of an unmodified AAC system, without fine-tuning for the new task. Our AAC system consists of an audio encoder (ConvNeXt-Tiny) trained on AudioSet for audio tagging, and a transformer decoder responsible for generating sentences. For AAC, it achieves a high SPIDEr-FL score of 0.298 on Clotho and 0.472 on AudioCaps on average. For ATR, we propose using the standard Cross-Entropy loss values obtained for any audio/caption pair. Experimental results on the Clotho and AudioCaps datasets demonstrate decent recall values using this simple approach. For instance, we obtained a Text-to-Audio R@1 value of 0.382 for AudioCaps, which is above the current state-of-the-art method without external data. Interestingly, we observe that normalizing the loss values was necessary for Audio-to-Text retrieval.

Domaines

Son [cs.SD]
Fichier principal
Vignette du fichier
KILLING_TWO_BIRDS_WITH_ONE_STONE__CAN_AN_AUDIO_CAPTIONING_SYSTEM_ALSO_BE_USED_FOR_AUDIO_TEXT_RETRIEVAL_.pdf (454.46 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04180972 , version 1 (28-08-2023)

Identifiants

  • HAL Id : hal-04180972 , version 1

Citer

Etienne Labbé, Thomas Pellegrini, Julien Pinquier. Killing two birds with one stone: can an audio captioning system also be used for audio-test retrieval?. 8th workshop on Detection and Classification of Acoustic Scenes and Events (DCASE2023), Sep 2023, Tampere, Finland. ⟨hal-04180972⟩
89 Consultations
39 Téléchargements

Partager

Gmail Facebook X LinkedIn More