Evaluating Reasoning vs. No‑Reasoning Models on Insider‑Threat Datasets

Sun, Lihao (2025) Evaluating Reasoning vs. No‑Reasoning Models on Insider‑Threat Datasets. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (796kB) | Preview
Preview	PDF (Configuration Manual) Download (746kB) | Preview

Abstract

Security operations centres (SOCs) are tasked with analyzing enormous volumes of asymmetrical log data that is unevenly distributed. Actual insider threat attacks are extremely hard to detect. This thesis asks one question: do “reasoning-oriented” LLMs (used as RL-post-trained / RL-distilled model families) work better than non-reasoning models for insider-threat detection on the CERT r6.2 dataset? We reformat raw log events into concise 4W session summaries (When, Where, What, Which), train models with careful attention to class imbalance data, and evaluate performance using AUPRC and AUROC. To keep training comparable, we adapt the 7B and 8B parameter models using LoRA with INT4 quantisation, fitting them onto consumer-grade GPUs typical in SOCs.

Our findings indicate that the ‘reasoning’ model—DeepSeek R1 Distill Qwen 7B with LoRA—achieves higher AUROC than the ‘no-reasoning’ model – LlaMA 3.1 8B Instruct with LoRA—on a 200k validation slice. However, baseline encoder models were evaluated on the full 1.41M-session validation set in the initial draft, which limits direct cross-model ranking. Therefore, conclusions are framed as protocol-specific until an apples-to-apples re-evaluation is completed on a fully harmonised evaluation set.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Mustafa, Raza Ul UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence Q Science > QA Mathematics > Computer software > Computer Security T Technology > T Technology (General) > Information Technology > Computer software > Computer Security
Divisions:	School of Computing > Master of Science in Cyber Security
Depositing User:	Ciara O'Brien
Date Deposited:	04 Sep 2026 11:16
Last Modified:	04 Sep 2026 11:16
URI:	https://norma.ncirl.ie/id/eprint/9837

Actions (login required)

View Item