A lightweight framework for detecting malicious traffic evasion via DNS over HTTPS

Karanwal, Surya Kant (2025) A lightweight framework for detecting malicious traffic evasion via DNS over HTTPS. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (1MB) | Preview
Preview	PDF (Configuration Manual) Download (2MB) | Preview

Abstract

Encrypted traffic of DNS over HTTPS (DoH) uses the same protocol that is used by normal web traffic, which makes it easy for an attacker to use this channel for data exfiltration or as a command and control channel (C2) without getting detected by traditional security devices. As almost all existing studies heavily rely on expensive and high computational solutions and leave a gap for a cost effective and easy to use solution. This study aims to solve this issue by building a lightweight rule-based solution for the detection of malicious traffic hiding in the network stream of DoH traffic. Additionally, this research also addresses the issue of detecting short lived traffic. The final artefact proposed in this study detects harmful DoH tunnels by analysing the patterns and behaviour of the flow using two types of rules, one of which is built using heuristics, such as flow time, packet count, TLS handshake behaviour, byte shares, etc, and the other is built by combining two detection layers, which are heuristics and JA3 signature/fingerprint to tackle both short lived and long lived traffic. As a result, the model was able to give a detection rate of 95% while tested with an individual type of rule, which was increased to 99% when both types of rules were combined. However, the model might need some tuning or adjustment for the detection of unseen malicious behaviour of the flow and TLS handshake.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Hafeez, Khadija UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > QA Mathematics > Computer software > Computer Security T Technology > T Technology (General) > Information Technology > Computer software > Computer Security
Divisions:	School of Computing > Master of Science in Cyber Security
Depositing User:	Ciara O'Brien
Date Deposited:	03 Sep 2026 11:11
Last Modified:	03 Sep 2026 11:11
URI:	https://norma.ncirl.ie/id/eprint/9800

Actions (login required)

View Item