A Knowledge-based, Data-driven Method for Action-sound Mapping - Archive ouverte HAL Access content directly
Conference Papers Year : 2017

A Knowledge-based, Data-driven Method for Action-sound Mapping


This paper presents a knowledge-based, data-driven method for using data describing action-sound couplings collected from a group of people to generate multiple complex mappings between the performance movements of a musician and sound synthesis. This is done by using a database of multimodal motion data collected from multiple subjects coupled with sound synthesis parameters. A series of sound stimuli is synthesised using the sound engine that will be used in performance. Multimodal motion data is collected by asking each participant to listen to each sound stimulus and move as if they were producing the sound using a musical instrument they are given. Multimodal data is recorded during each performance, and paired with the synthesis parameters used for generating the sound stimulus. The dataset created using this method is then used to build a topological representation of the performance movements of the subjects. This representation is then used to interactively generate training data for machine learning algorithms, and define mappings for real-time performance. To better illustrate each step of the procedure, we describe an implementation involving clarinet, motion capture, wearable sensor armbands, and waveguide synthesis.
Fichier principal
Vignette du fichier
paper0043.pdf (1.05 Mo) Télécharger le fichier
Origin Publisher files allowed on an open archive

Dates and versions

hal-01577885 , version 1 (22-07-2019)


  • HAL Id : hal-01577885 , version 1


Federico Visi, Baptiste Caramiaux, Michael Mcloughlin. A Knowledge-based, Data-driven Method for Action-sound Mapping. NIME 2017 : New Interfaces for Musical Expression, May 2017, Copenhagen, Denmark. ⟨hal-01577885⟩
170 View
52 Download


Gmail Mastodon Facebook X LinkedIn More