Binaural Sound Localization in Noisy Environments Using Frequency-Based Audio Vision Transformer (FAViT) - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Binaural Sound Localization in Noisy Environments Using Frequency-Based Audio Vision Transformer (FAViT)

Résumé

Binaural sound source localization (BSSL) aims to locate sound as the way human does, but it falls short due to acoustic interferences. While Convolutional Neural Networks (CNNs) have shown promise in localizing sounds corrupted by noise, their large parameter and training data requirements make them unsuitable for real-time processing on devices like hearing aids and robots. In this paper, we propose an adapted Vision Transformer (ViT) model for BSSL in noisy environments. Inspired by the Duplex Theory, our model uses selective attention mechanisms to the frequency range of binaural features to aid in sound localization. Our model outperformed recent CNNs and standard audio ViT models in localizing speech in unseen noises and speakers, even in challenging conditions with low training data and parameters. The attention heatmap results suggest differences in how humans and machines process binaural cues, opening up for further investigation.
Fichier principal
Vignette du fichier
Phokhinanan2023.pdf (397.67 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-04416143 , version 1 (25-01-2024)

Identifiants

Citer

Waradon Phokhinanan, Nicolas Obin, Sylvain Argentieri. Binaural Sound Localization in Noisy Environments Using Frequency-Based Audio Vision Transformer (FAViT). INTERSPEECH, Aug 2023, Dublin, Ireland. pp.3704-3708, ⟨10.21437/Interspeech.2023-2015⟩. ⟨hal-04416143⟩
31 Consultations
33 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More