MusicLDM: Enhancing Novelty in text-to-music Generation Using Beat-Synchronous mixup Strategies - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

MusicLDM: Enhancing Novelty in text-to-music Generation Using Beat-Synchronous mixup Strategies

Yusong Wu
  • Fonction : Auteur
Haohe Liu
  • Fonction : Auteur
Marianna Nezhurina
  • Fonction : Auteur
Taylor Berg-Kirkpatrick
  • Fonction : Auteur
Shlomo Dubnov
  • Fonction : Auteur

Résumé

Diffusion models have shown promising results in cross-modal generation tasks, including text-to-image and text-to-audio generation. However, generating music, as a special type of audio, presents unique challenges due to limited availability of music data and sensitive issues related to copyright and plagiarism. In this paper, to tackle these challenges, we first construct a state-of-theart text-to-music model, MusicLDM, that adapts Stable Diffusion and AudioLDM architectures to the music domain. Then, to address the limitations of training data and to avoid plagiarism, we leverage a beat tracking model and propose two different mixup strategies for data augmentation: beat-synchronous audio mixup and beat-synchronous latent mixup, which recombine training audio directly or via a latent embeddings space, respectively. Such mixup strategies encourage the model to interpolate between musical training samples and generate new music within the convex hull of the training data, making the generated music more diverse while still staying faithful to the corresponding style. In addition to popular evaluation metrics, we design several new evaluation metrics based on CLAP score to demonstrate that our proposed MusicLDM and beat-synchronous mixup strategies improve both the quality and novelty of generated music, as well as the correspondence between input text and generated music.
Fichier principal
Vignette du fichier
MusicLDM___ICASSP_2024.pdf (5.2 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04766515 , version 1 (05-11-2024)

Identifiants

Citer

Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, et al.. MusicLDM: Enhancing Novelty in text-to-music Generation Using Beat-Synchronous mixup Strategies. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr 2024, Seoul, France. pp.1206-1210, ⟨10.1109/ICASSP48485.2024.10447265⟩. ⟨hal-04766515⟩
2 Consultations
4 Téléchargements

Altmetric

Partager

More