Although extensively studied for many years, defining the timbre of musical sounds remains unclear and somewhatcontroversial. We here address this question by using representations of sounds inspired by auditory cortical processes - so-called spectro-temporal modulation representations - as front-end representations to interpretable metric learning techniques modelling human dissimilarity ratings. We present a meta-analysis of 17 published experiments on the perception of musical instrument timbre. The results reveal that these studies are only partly replicable. Interestingly, we observed that spectro-temporal modulations embed relevant information to model human dissimilarity ratings of musical instruments sounds. Thanks to an interpretable distance metric learning technique, the results strikingly suggest that humans use both generic and context-driven acoustical cues defining the different facets of musical instrument timbre. This study hereby provides a unique overview of 17 historical studies on timbre and points the limitation of the traditional dimensional analyses. We further propose a new way to investigate the acoustical correlates of timbre. The proposed methodology hence opens avenues to link acoustical representations to high-level human judgements.