Enhanced Fileless Malware Detection in Memory using Convolutional Neural Networks

Ajayi, Ademola Temitayo (2025) Enhanced Fileless Malware Detection in Memory using Convolutional Neural Networks. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (884kB) | Preview
Preview	PDF (Configuration Manual) Download (1MB) | Preview

Abstract

This thesis report looks at why spotting fileless malware’s getting tougher as it runs only in computer memory, uses real Windows tools like PowerShell or Windows Management Instrumentation (WMI), plus it slips past regular antivirus by avoiding files. Traditional-based detection miss these hidden attacks because they rely on known patterns and so we need new ways to focus on what programs do and how memory gets used. To fix this gap, the thesis builds a CNN model trained on live-memory artefacts pulled from the CIC-MalMem-2022 data collection and it comprises of information like running processes, open handles, loaded libraries, modules, signs of code injections e.t.c all gathered via Volatility plugins such as pslist, malfind, ldrmodules e.t.c. The data gets cleaned up, scaled evenly, then packed into 8×8 matrices that help the neural network see attack habits based on spatial layout. The CNN's performance is tested thoroughly, compared side by side with four standard classifiers which are Random Forest, Naïve Bayes, RBF-SVM, and XGBoost by using measures like accuracy, precision, recall, F1-score, and ROC-AUC. Results show almost flawless detection for every model; XGBoost hits full marks across the board, while the suggested CNN scores 99.92% in accuracy and 99.98% in ROC-AUC, showing it handles unseen data well and rarely gives false positives. Though ensemble methods edge out CNNs slightly in numbers, CNNs win by automatically building their own features from raw input. That cuts down on manual feature engineering plus makes them more flexible when malware change tactics. The outcomes back up CNN-based memory artefact modelling as a robust foundation for next-generation, in-memory fileless malware detection, opening doors to explore time-aware neural networks and real-world endpoint telemetry.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Pantridge, Michael UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > QA Mathematics > Computer software > Computer Security T Technology > T Technology (General) > Information Technology > Computer software > Computer Security Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Machine learning
Divisions:	School of Computing > Master of Science in Cyber Security
Depositing User:	Ciara O'Brien
Date Deposited:	03 Sep 2026 10:07
Last Modified:	03 Sep 2026 10:07
URI:	https://norma.ncirl.ie/id/eprint/9787

Actions (login required)

View Item