AutoRoC-DBSCAN: Automatic tuning of DBSCAN to detect malicious DNS tunnels - Archive ouverte HAL Access content directly
Journal Articles Annals of Telecommunications - annales des télécommunications Year : 2024

AutoRoC-DBSCAN: Automatic tuning of DBSCAN to detect malicious DNS tunnels

Abstract

Modern attacks, such as advanced persistent threats, hide command-and-control channels inside authorized network traffic like DNS or DNS over HTTPS to infiltrate the local network and exfiltrate sensitive data. Detecting such malicious traffic using traditional techniques is cumbersome especially when the traffic encrypted like DNS over HTTPS. Unsupervised machine learning techniques, and more specifically density-based spatial clustering of applications with noise (DBSCAN), can achieve good results in detecting malicious DNS tunnels. However, DBSCAN requires manually tuning two hyperparameters, whose optimal values can differ depending on the dataset. In this article, we propose an improved algorithm called AutoRoC-DBSCAN that can automatically find the best hyperparameters. We evaluated and obtained good results on two different datasets: a dataset we created with malicious DNS tunnels and the CIRA-CIC-DoHBrw-2020 dataset with malicious DoH tunnels.
No file

Dates and versions

hal-04522675 , version 1 (26-03-2024)

Identifiers

Cite

Thi Quynh Nguyen, Romain Laborde, Abdelmalek Benzekri, Arnaud Oglaza, Mehdi Mounsif. AutoRoC-DBSCAN: Automatic tuning of DBSCAN to detect malicious DNS tunnels. Annals of Telecommunications - annales des télécommunications, inPress, ⟨10.1007/s12243-024-01025-5⟩. ⟨hal-04522675⟩
30 View
0 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More