In our work on computational design of expressive gestural interaction, we experienced various challenges for advanced optimisation methods. Here we want to highlight two of these challenges based on the design and the use of a Bayesian model called Gesture Variation Follower, with the aim to discuss such challenges with a broader community of designers and HCI practitioners during the workshop.