Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizer - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizer

Résumé

In this work, we propose to study the performance of a model trained with a sentence embedding regression loss component for the Automated Audio Captioning task. This task aims to build systems that can describe audio content with a single sentence written in natural language. Most systems are trained with the standard Cross-Entropy loss, which does not take into account the semantic closeness of the sentence. We found that adding a sentence embedding loss term reduces overfitting, but also increased SPIDEr from 0.397 to 0.418 in our first setting on the AudioCaps corpus. When we increased the weight decay value, we found our model to be much closer to the current state-of-the-art scores, with a SPIDEr score up to 0.444 compared to a 0.475 score. Moreover, this model uses eight times less trainable parameters than the current state-of-the-art method Multi-TTA. In this training setting, the sentence embedding loss has no more impact on the model performance.
Fichier principal
Vignette du fichier
Multitask_learning_in_Audio_Captioning__a_sentence_embedding_regression_loss_acts_as_a_regularizer.pdf (371.81 Ko) Télécharger le fichier
img/SBERT-Loss-V2a.pdf (45.47 Ko) Télécharger le fichier
img/SBERT-Loss-V2b.pdf (41.19 Ko) Télécharger le fichier
img/val_loss_over_epoch-optim_AdamW-wd_1e-06-smooth_0.0.pdf (29.96 Ko) Télécharger le fichier
img/val_loss_over_epoch-optim_AdamW-wd_2.0-smooth_0.0.pdf (29.91 Ko) Télécharger le fichier
img/val_sbert.sim_over_epoch-optim_AdamW-wd_1e-06-smooth_0.75.pdf (29.89 Ko) Télécharger le fichier
img/val_sbert.sim_over_epoch-optim_AdamW-wd_2.0-smooth_0.75.pdf (29.87 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Origine : Fichiers produits par l'(les) auteur(s)
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04207519 , version 1 (14-09-2023)

Licence

Paternité

Identifiants

Citer

Etienne Labbé, Julien Pinquier, Thomas Pellegrini. Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizer. 31st European Signal Processing Conference (EUSIPCO 2023), Sep 2023, Helsinki, Finland. ⟨10.48550/arXiv.2305.01482⟩. ⟨hal-04207519⟩
42 Consultations
21 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More