Mahajan, Sanket Shrikrishna (2025) Enhancing Detection of Encrypted Fileless Malware using Memory Forensics and AI Techniques. Masters thesis, Dublin, National College of Ireland.
Preview |
PDF (Master of Science)
Download (783kB) | Preview |
Preview |
PDF (Configuration Manual)
Download (907kB) | Preview |
Abstract
Fileless malware holds a significant challenge to cybersecurity because it does not touch a disk. It cannot be detected by traditional file-based malware detectors. This paper proposes a hybrid detection system with memory forensics answers, entropy profiling, rule-based detection and un-supervised AI clustering to identify miraculous and apparently encrypted fileless exercises in progress. To find out the structured memory artifacts that were acquired by Volatility, the system incorporates them, and it analyzes some essential measurements of the behavior, such as process injection rates, system callback frequencies, driver execution, and Shannon entropy. A rule-based engine assigns levels of severity and matches the behavior with MITRE ATT&CK methods, and entropy profiling determines activities with high degrees of randomness that could indicate encryption. Also, the AI model is implemented using Principal Component Analysis and K-Means clustering to expose hidden anomalies in memory activity. The results are summarized into reports, CSV, and graphs that make reports understood. The result of the findings displays the ability of the hybrid approach to retrieve high-risk procedures and distinct grouping of anomalous acts in the PCA space. The visualizations of results were conducted and statistically compared with actual forensic data which were created through modular Python-based scripts. This pipeline provides a scalable method in the triaging of malware and the inspection in real-time on memory-based threats that can be understood.
Actions (login required)
![]() |
View Item |
Tools
Tools