From Sound to Sight: Audio-Visual Fusion and Deep Learning for Drone Detection - Archive ouverte HAL Access content directly
Conference Papers Year : 2024

From Sound to Sight: Audio-Visual Fusion and Deep Learning for Drone Detection

Ildi Alla
  • Function : Author
  • PersonId : 1371092
Valeria Loscri
  • Function : Author
  • PersonId : 1084587
Marco Levorato
  • Function : Author
  • PersonId : 1371094


The proliferation of airborne drones, while instrumental to a broad range of applications, has led to an increased number of regulatory non-compliance incidents. The ubiquitous unmanned aerial vehicles (UAVs) are posing security risks, since they have started to be used for cybercrimes. Effective detection of illicit drones in restricted areas is paramount. Evolved drones are more and more sophisticated, and sometimes they do not emit RF-based signals, making inapplicable RF-based detection solutions. Different from existing work, this paper introduces a neural sensor fusion framework for drone detection based on both audio and video data to accurately identify drones and differentiate them from similar objects at long distances. Our design adopts a late fusion approach using the Weighted Average and Random Forest algorithm on the visual and auditory classification pipeline. Specifically, we process infrared data using a You Only Look Once (YOLO) v5 model due to its balance between inference time and accuracy. For the audio stream, we evaluate Long Short-Term Memory (LSTM) and Convolutional Recurrent Neural Network (CRNN) models and demonstrate the superiority of the CRNN model through Mel-Frequency Cepstral Coefficients (MFCC) features. To demonstrate the robustness of our audio-visual fusion approach, we validate it in extensive scenarios, with impaired audio/video data. Our results demonstrate that multimodal fusion significantly improves drone detection, outperforming traditional single-modality systems in complex environments. Additionally, our system provides predictions rapidly, in just 0.382 seconds, making it well-suited for real-time applications.
Fichier principal
Vignette du fichier
From_Sound_to_Sight_HAL_Version.pdf (10.91 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04532239 , version 1 (04-04-2024)



Ildi Alla, Hervé B Olou, Valeria Loscri, Marco Levorato. From Sound to Sight: Audio-Visual Fusion and Deep Learning for Drone Detection. WiSec ’24, May 27–30, 2024, Seoul, Republic of Korea, May 2024, Seoul (Korea), South Korea. ⟨10.1145/3643833.3656133⟩. ⟨hal-04532239⟩


169 View
208 Download



Gmail Mastodon Facebook X LinkedIn More