Toward a realistic model of speech processing in the brain with self-supervised learning - Archive ouverte HAL Access content directly
Conference Papers Year : 2022

Toward a realistic model of speech processing in the brain with self-supervised learning

Pierre Orhan
  • Function : Author
  • PersonId : 1172918
Yves Boubenec


Several deep neural networks have recently been shown to generate activations similar to those of the brain in response to the same input. These algorithms, however, remain largely implausible: they require extraordinarily large amounts of data, unobtainable supervised labels, textual rather than raw sensory input, and / or implausibly large memory (e.g. thousands of contextual words). These elements highlight the need to identify algorithms that, under these limitations, would suffice to account for both behavioral and brain responses. Focusing on speech processing, we here hypothesize that self-supervised algorithms trained on the raw waveform constitute a promising candidate. Specifically, we compare a recent self-supervised model, wav2vec 2.0, to the brain activity of 412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to approximately one hour of audio books. First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech-a quantity comparable to what infants can be exposed to during language acquisition. Second, its functional hierarchy aligns with the cortical hierarchy of speech processing. Third, different training regimes reveal a functional specialization akin to the cortex: wav2vec 2.0 learns sound-generic, speech-specific and language-specific representations similar to those of the prefrontal and temporal cortices. Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants. These elements, resulting from the largest neuroimaging benchmark to date, show how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.
Fichier principal
Vignette du fichier
_neurips_2022__Charlotte_Juliette (4).pdf (4.22 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03808200 , version 1 (10-10-2022)




  • HAL Id : hal-03808200 , version 1


Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, et al.. Toward a realistic model of speech processing in the brain with self-supervised learning. NeurIPS 2022 - 36th Conference on Neural Information Processing Systems, Nov 2022, New Orleans, United States. ⟨hal-03808200⟩
62 View
73 Download


Gmail Facebook X LinkedIn More