Multivariate pattern analysis (MVPA) of functional neuroimaging data has emerged as a key tool for studying the cognitive architecture of the human brain. At the group level, we have recently demonstrated the advantages of an under-exploited scheme that consists in training a machine learning model on data from a set of subjects and evaluating its generalization ability on data from unseen subjects (see Inter-subject pattern analysis: A straightforward and powerful scheme for group-level MVPA [1]). We here provide a data set that is fully ready to perform inter-subject pattern analysis, which includes 5616 single-trial brain activation maps recorded in 39 participants who were scanned using functional magnetic resonance imaging (fMRI) with a voice localizer paradigm. This data set should therefore reveal valuable for data scientists developing brain decoding algorithms as well as cognitive neuroscientists interested in voice perception.