NORMA eResearch @NCI Library

Enhancing Detection of Encrypted Fileless Malware using Memory Forensics and AI Techniques

Mahajan, Sanket Shrikrishna (2025) Enhancing Detection of Encrypted Fileless Malware using Memory Forensics and AI Techniques. Masters thesis, Dublin, National College of Ireland.

[thumbnail of Master of Science]
Preview
PDF (Master of Science)
Download (783kB) | Preview
[thumbnail of Configuration Manual]
Preview
PDF (Configuration Manual)
Download (907kB) | Preview

Abstract

Fileless malware holds a significant challenge to cybersecurity because it does not touch a disk. It cannot be detected by traditional file-based malware detectors. This paper proposes a hybrid detection system with memory forensics answers, entropy profiling, rule-based detection and un-supervised AI clustering to identify miraculous and apparently encrypted fileless exercises in progress. To find out the structured memory artifacts that were acquired by Volatility, the system incorporates them, and it analyzes some essential measurements of the behavior, such as process injection rates, system callback frequencies, driver execution, and Shannon entropy. A rule-based engine assigns levels of severity and matches the behavior with MITRE ATT&CK methods, and entropy profiling determines activities with high degrees of randomness that could indicate encryption. Also, the AI model is implemented using Principal Component Analysis and K-Means clustering to expose hidden anomalies in memory activity. The results are summarized into reports, CSV, and graphs that make reports understood. The result of the findings displays the ability of the hybrid approach to retrieve high-risk procedures and distinct grouping of anomalous acts in the PCA space. The visualizations of results were conducted and statistically compared with actual forensic data which were created through modular Python-based scripts. This pipeline provides a scalable method in the triaging of malware and the inspection in real-time on memory-based threats that can be understood.

Item Type: Thesis (Masters)
Supervisors:
Name
Email
Heffernan, Niall
UNSPECIFIED
Uncontrolled Keywords: Fileless Malware; Memory Forensics; Volatility; Entropy Profiling; Rule-Based Detection; MITRE ATT&CK; PCA; K-Means Clustering; Hybrid Detection System
Subjects: Q Science > QA Mathematics > Electronic computers. Computer science
T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science
Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence
Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence
Q Science > QA Mathematics > Computer software > Computer Security
T Technology > T Technology (General) > Information Technology > Computer software > Computer Security
Divisions: School of Computing > Master of Science in Cyber Security
Depositing User: Ciara O'Brien
Date Deposited: 18 Aug 2026 16:50
Last Modified: 18 Aug 2026 16:50
URI: https://norma.ncirl.ie/id/eprint/9541

Actions (login required)

View Item View Item