SingSong: Generating musical accompaniments from singing - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

SingSong: Generating musical accompaniments from singing

Chris Donahue
  • Fonction : Auteur
Antoine Caillon
  • Fonction : Auteur
Adam Roberts
  • Fonction : Auteur
Ethan Manilow
  • Fonction : Auteur
Andrea Agostinelli
  • Fonction : Auteur
Mauro Verzetti
  • Fonction : Auteur
Ian Simon
  • Fonction : Auteur
Olivier Pietquin
  • Fonction : Auteur
Neil Zeghidour
  • Fonction : Auteur
Jesse Engel
  • Fonction : Auteur

Résumé

We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Specifically, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022) -- a state-of-the-art approach for unconditional audio generation -- to be suitable for conditional "audio-to-audio" generation tasks, and train it on the source-separated (vocal, instrumental) pairs. In a pairwise comparison with the same vocal inputs, listeners expressed a significant preference for instrumentals generated by SingSong compared to those from a strong retrieval baseline. Sound examples at https://g.co/magenta/singsong

Dates et versions

hal-04467269 , version 1 (20-02-2024)

Identifiants

Citer

Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, et al.. SingSong: Generating musical accompaniments from singing. International Conference on Machine Learning ICML 2023, 2023, Hawaii, United States. ⟨hal-04467269⟩
12 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More