Spectrogram Inpainting for Interactive Generation of Instrument Sounds - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Spectrogram Inpainting for Interactive Generation of Instrument Sounds

Théis Bazin
  • Fonction : Auteur
Gaëtan Hadjeres
  • Fonction : Auteur
Mikhail Malt

Résumé

Modern approaches to sound synthesis using deep neural networks are hard to control, especially when fine-grained conditioning information is not available, hindering their adoption by musicians. In this paper, we cast the generation of individual instrumental notes as an inpainting-based task, introducing novel and unique ways to iteratively shape sounds. To this end, we propose a two-step approach: first, we adapt the VQ-VAE-2 image generation architecture to spectrograms in order to convert real-valued spectrograms into compact discrete codemaps, we then implement token-masked Transformers for the inpainting-based generation of these codemaps. We apply the proposed architecture on the NSynth dataset on masked resampling tasks. Most crucially, we open-source an interactive web interface to transform sounds by inpainting, for artists and practitioners alike, opening up to new, creative uses.

Dates et versions

hal-03207968 , version 1 (26-04-2021)

Identifiants

Citer

Théis Bazin, Gaëtan Hadjeres, Philippe Esling, Mikhail Malt. Spectrogram Inpainting for Interactive Generation of Instrument Sounds. Proceedings of the 2020 Joint Conference on AI Music Creativity, Jul 2020, Stockholm, Norway. ⟨10.30746/978-91-519-5560-5⟩. ⟨hal-03207968⟩
27 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More