Enhancing the Robustness of AI-Based Malware Detection: A Hybrid Approach Combining Static and Dynamic Analysis to Defend Against Adversarial Evasion Attacks

Upasani, Shreya Mangesh (2025) Enhancing the Robustness of AI-Based Malware Detection: A Hybrid Approach Combining Static and Dynamic Analysis to Defend Against Adversarial Evasion Attacks. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (1MB) | Preview
Preview	PDF (Configuration Manual) Download (276kB) | Preview

Abstract

Artificial Intelligence (AI) has significantly advanced the field of malware detection, enabling models to recognise complex patterns from large datasets. Yet adversarial evasion where attackers make subtle changes to features without affecting the malicious functionality still remains a major challenge. This research introduces a hybrid detection framework that combines static features from the EMBER 2018 dataset with dynamic features from the CIC-MalMem-2022 dataset, supplemented where necessary with synthetic benign dynamic samples. Rather than performing live sandboxing, the study relied on pre-extracted behavioural features.

The framework was evaluated in three configurations: static-only, dynamic-only and hybrid. Logistic Regression, Random Forest and XGBoost were applied across all three, with a Multi-Layer Perceptron (MLP) added specifically for the hybrid case. Balanced subsets of 200 samples were used for each configuration (100 benign and 100 malicious) to maintain class balance and computational feasibility. Adversarial robustness was tested in two ways: targeted feature perturbations on high-impact attributes for all models and a gradient-based Fast Gradient Sign Method (FGSM) attack on the hybrid MLP. Results showed that the hybrid Random Forest achieved the highest resilience, with only minimal accuracy loss under attack, outperforming the static-only and dynamic-only approaches. These findings highlight that fusing static and dynamic features can be an effective strategy for strengthening malware detection against adversarial manipulation.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Salahuddin, Jawad UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence Q Science > QA Mathematics > Computer software > Computer Security T Technology > T Technology (General) > Information Technology > Computer software > Computer Security
Divisions:	School of Computing > Master of Science in Cyber Security
Depositing User:	Ciara O'Brien
Date Deposited:	19 Aug 2026 15:47
Last Modified:	19 Aug 2026 15:47
URI:	https://norma.ncirl.ie/id/eprint/9557

Actions (login required)

View Item