The proliferation of airborne drones, while instrumental to a broad range of applications, has led to an increased number of regulatory non-compliance incidents. The ubiquitous unmanned aerial vehicles (UAVs) are posing security risks, since they have started to be used for cybercrimes. Effective detection of illicit drones in restricted areas is paramount. Evolved drones are more and more sophisticated, and sometimes they do not emit RF-based signals, making inapplicable RF-based detection solutions. Different from existing work, this paper introduces a neural sensor fusion framework for drone detection based on both audio and video data to accurately identify drones and differentiate them from similar objects at long distances. Our design adopts a late fusion approach using the Weighted Average and Random Forest algorithm on the visual and auditory classification pipeline. Specifically, we process infrared data using a You Only Look Once (YOLO) v5 model due to its balance between inference time and accuracy. For the audio stream, we evaluate Long Short-Term Memory (LSTM) and Convolutional Recurrent Neural Network (CRNN) models and demonstrate the superiority of the CRNN model through Mel-Frequency Cepstral Coefficients (MFCC) features. To demonstrate the robustness of our audio-visual fusion approach, we validate it in extensive scenarios, with impaired audio/video data. Our results demonstrate that multimodal fusion significantly improves drone detection, outperforming traditional single-modality systems in complex environments. Additionally, our system provides predictions rapidly, in just 0.382 seconds, making it well-suited for real-time applications.