Detecting abnormal DNS traffic using unsupervised machine learning

Thi Quynh Nguyen (1, 2) , Romain Laborde (1) , Abdelmalek Benzekri (1, 3) , Bruno Qu’hen (2)

1 IRIT-SIERA - Service IntEgration and netwoRk Administration
2 MODIS Toulouse
3 UT3 - Université Toulouse III - Paul Sabatier

Nowadays, complex attacks like Advanced Persistent Threats (APTs) often use tunneling techniques to avoid being detected by security systems like Intrusion Detection System (IDS), Security Event Information Management (SIEMs) or firewalls. Companies try to identify these APTs by defining rules on their intrusion detection system, but it is a hard task that requires a lot of time and effort. In this study, we compare the performance of four unsupervised machine-learning algorithms: K-means, Gaussian Mixture Model (GMM), Density-Based Spatial Clustering of Applications with Noise (DBSCAN), and Local Outlier Factor (LOF) on the Boss of the SOC Dataset Version 1 (Botsv1) dataset of the Splunk project to detect malicious DNS traffics. Then we propose an approach that combines DBSCAN and K Nearest Neighbor (KNN) to achieve 100% detection rate and between 1.6% and 2.3% false-positive rate. A simple post-analysis consisting in ranking the IP addresses according to the number of requests or volume of bytes sent determines the infected machines.

Mots clés: Anomaly detection; DNS tunneling; C&C; K-means; GMM; LOF; DBSCAN; Clustering algorithms

Domaines: Cryptographie et sécurité [cs.CR]; Réseaux et télécommunications [cs.NI]

Thi Quynh Nguyen, Romain Laborde, Abdelmalek Benzekri, Bruno Qu’hen. Detecting abnormal DNS traffic using unsupervised machine learning. 4th Cyber Security in Networking Conference: Cyber Security in Networking (CSNet 2020), IEEE Communications Society, Oct 2020, Lausanne, Switzerland. pp.1-8, ⟨10.1109/CSNet50428.2020.9265466⟩. ⟨hal-03184957⟩