AutoRoC-DBSCAN: Automatic Tuning of DBSCAN to Detect Malicious DNS Tunnels
Résumé
Domain Name System (DNS) is one of the core services of the Internet. Whereas it was created to associate domain names and IP addresses, modern attacks, such as Advanced Persistent Threats, use tunneling techniques over the DNS protocol to build covert channels in order to infiltrate the local network or exfiltrate data while bypassing security systems. Unsupervised machine learning techniques, and more specifically, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), can achieve good results in detecting simple malicious DNS tunnels. However, DBSCAN requires manually tuning two hyperparameters, whose optimal values can be different depending on the dataset. In this article, we propose an improved algorithm called AutoRoC-DBSCAN that can automatically find the best hyperparameters, in order to automate the detection of malicious DNS tunnels in a completely new dataset.