A novel Deep Q-Learning algorithm for anomaly detection

Abrahani, Ammar Yousuf (2025) A novel Deep Q-Learning algorithm for anomaly detection. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (1MB) | Preview
Preview	PDF (Configuration Manual) Download (750kB) | Preview

Abstract

The detection of anomalies is essential to identify unexpected patterns that can indicate fraud, security threats, or system failures. Classical models often depend on predefined rules or labelled data and become ineffective in dynamic or imbalanced environments. This research addresses these limitations by comparing advanced machine learning approaches, Deep Autoencoders and Deep Q-Learning (DQL) with classical models such as Isolation Forest and Local Outlier Factor (LOF).

Experiments were conducted on synthetic datasets generated with scikit-learn and the Kaggle Credit Card Fraud dataset. Isolation Forest emerged as a stronger classical baseline, achieving a macro-average F1-score of 0.934 on structured data. The Deep Autoencoder achieved a macro-average F1-score of 0.922, outperforming wide and shallow variants, while the DQL model demonstrated progressive improvement across 20 episodes, converging at approximately 0.92. However, in the imbalanced Kaggle dataset, both deep and classical methods experienced performance degradation (Autoencoder: 0.49) reflecting challenges in generalisation under extreme imbalance.

Unlike previous surveys that conceptually review reinforcement learning for anomaly detection, this study provides empirical benchmarking of reinforcement learning against both classical and neural models under identical conditions. The findings highlight that while deep and reinforcement learning approaches outperform static methods on structured data, their robustness diminishes with atypical distributions. This underscores the need for hybrid methods, temporal modelling, and hyperparameter optimisation to enhance real-world applicability.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Estrada, Giovani UNSPECIFIED
Subjects:	T Technology > T Technology (General) > Information Technology > Cloud computing Q Science > QA Mathematics > Computer software > Computer Security T Technology > T Technology (General) > Information Technology > Computer software > Computer Security Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Machine learning
Divisions:	School of Computing > Master of Science in Cloud Computing
Depositing User:	Ciara O'Brien
Date Deposited:	20 Mar 2026 09:50
Last Modified:	20 Mar 2026 09:50
URI:	https://norma.ncirl.ie/id/eprint/9194

Actions (login required)

View Item