Explainable Large Language Model (LLM)-Based Detection of Advanced Social Engineering Tactics in Phishing Emails: AI - Advanced Detection of Textual Employed SE with XAI

Monaghan, Gary (2025) Explainable Large Language Model (LLM)-Based Detection of Advanced Social Engineering Tactics in Phishing Emails: AI - Advanced Detection of Textual Employed SE with XAI. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (667kB) | Preview
Preview	PDF (Configuration Manual) Download (217kB) | Preview

Abstract

Phishing emails remain on of the most significant threats regarding information security, often employing sophisticated social engineering techniques which evade modern, traditional detection systems, which fail to identify and subsequently limit response efforts to mitigate their potential effects. Significant improvements in Natural Language Processing (NLP), particularly seen with transformer-based models, such as DistilBERT, RoBERTa and GPT-Neo, show promise when tasked with detecting these nuanced threats. Nevertheless, the effective training of models requires extensive rea-world datasets, which are often limited by privacy, ethical and logistical constraints. As a result of this, security researchers must rely on synthetically generated datasets, which lack in variability, subtlety and the overall realism of real-world phishing attacks. This project evaluates the performance of transformer-based models that have been trained on a sizable synthetically generated dataset, purpose created for this project. Focused on assessing the generalisability, robustness and interpretability of the models. For each model, they were assessed based of baseline parameters, a set of enhanced/optimised parameters and finally with sentence-level tokenisation implemented. Training was performed rigorously and then systematically evaluated against real-world collected phishing datasets. Enhanced parameter tuning consistently improved the model’s accuracy and robustness. Sentence-level tokenisation resulted in mixed performance outcomes, with results that either enhanced or impeded classification efficacy. Explainable Artificial Intelligence (XAI) methods, Local Interpretable Model-Agnostic Explanations (LIME) and Shapley Additive exPlanations (SHAP), were trialled in order to provide insights into model decisions, in an effort to aid user understanding and aid security response teams in their efforts to mitigate and better understand socially engineered phishing attacks. Impressive overall performance metrics were observed, however, further evaluations indicate vulnerabilities exist when tasked with detecting minor linguistic manipulations, such as synonym substitutions, highlighting limitations inherent when using synthetic datasets. Ultimately, this project provides insights into the improvement of transformer-based phishing detection, whilst advocating for the integration of real-world phishing datasets and continuous learning strategies to enhance the effectiveness, generalisability and interpretability of phishing solutions.

Item Type:	Thesis (Masters)
Supervisors:	Name Email -, - UNSPECIFIED
Subjects:	Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence P Language and Literature > P Philology. Linguistics > Computational linguistics. Natural language processing Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Machine learning
Divisions:	School of Computing > Master of Science in Artificial Intelligence
Depositing User:	Ciara O'Brien
Date Deposited:	12 Aug 2026 09:11
Last Modified:	12 Aug 2026 09:11
URI:	https://norma.ncirl.ie/id/eprint/9509

Actions (login required)

View Item