The cerebral processing of voice information is known to engage Temporal Voice Areas (TVAs) that respond preferentially to conspecific vocalizations. But how voice information related to the stable physical characteristics of the speaker such as gender, age or identity is represented by neuronal populations in these areas remains poorly understood. Here we used a deep neural network (DNN) to generate a high-level, small-dimension representational space of voice stimuli -the 'voice latent space' (VLS)- and examined its linear relation with cerebral activity via encoding, representational similarity and decoding analyses. We find that the VLS maps onto fMRI measures of cerebral activity in response to tens of thousands of voice stimuli from hundreds of different speaker identities, and better accounts for the representational geometry for speaker identity in the TVAs than in A1. Moreover, the VLS allowed TVA-based reconstructions of voice stimuli that preserved important aspects of speaker gender and identity as assessed by both machine classifiers and human listeners. These results demonstrate that a low-dimensional, DNN-derived space accounts well for cerebral voice representations and provide insights into representational differences between A1 and the TVAs, paving the way to noninvasive brain-computer interface applications.