Retrieval Guided Music Captioning via Multimodal Prefixes - Archive ouverte HAL
Communication Dans Un Congrès Année : 2023

Retrieval Guided Music Captioning via Multimodal Prefixes

Nikita Srivatsan
  • Fonction : Auteur
  • PersonId : 1436190
Ke Chen
  • Fonction : Auteur
Shlomo Dubnov
Taylor Berg-Kirkpatrick
  • Fonction : Auteur

Résumé

In this paper we put forward a new approach to music captioning, the task of automatically generating natural language descriptions for songs. These descriptions are useful both for categorization and analysis, and also from an accessibility standpoint as they form an important component of closed captions for video content. Our method supplements an audio encoding with a retriever, allowing the decoder to condition on multimodal signal both from the audio of the song itself as well as a candidate caption identified by a nearest neighbor system. This lets us retain the advantages of a retrieval based approach while also allowing for the flexibility of a generative one. We evaluate this system on a dataset of 200k music-caption pairs scraped from Audiostock, a royalty-free music platform, and on MusicCaps, a dataset of 5.5k pairs. We demonstrate significant improvements over prior systems across both automatic metrics and human evaluation.

Domaines

Son [cs.SD]
Fichier principal
Vignette du fichier
ijcai_music_captioning.pdf (378.44 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04766522 , version 1 (05-11-2024)

Identifiants

Citer

Nikita Srivatsan, Ke Chen, Shlomo Dubnov, Taylor Berg-Kirkpatrick. Retrieval Guided Music Captioning via Multimodal Prefixes. Thirty-Third International Joint Conference on Artificial Intelligence {IJCAI-24}, Aug 2023, Jeju, South Korea. pp.7762-7770, ⟨10.24963/ijcai.2024/859⟩. ⟨hal-04766522⟩

Altmetric

Partager

More