Real-time anomaly detection in users’ shell command histories using TextCNN

Ravi, Surya (2025) Real-time anomaly detection in users’ shell command histories using TextCNN. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (1MB) | Preview
Preview	PDF (Configuration Manual) Download (1MB) | Preview

Abstract

Linux command-line environments continue to be a popular target for attackers to escalate their privileges, steal data, and gain remote access through stealthy command sequences. Conventional detection methods rely on signature based or on predefined rules, and hence they fail to detect new, obfuscated, or attacker-crafted shell commands. Motivated by this limitation, this study aims to identify whether a lightweight deep-learning model can accurately distinguish benign or malicious shell commands in real time on a Linux environment without relying on kernel-level instrumentation. A supervised TextCNN classifier was developed and trained using a combined dataset of real-world shell commands and manually crafted malicious examples executed within an isolated RHEL virtual machine. Commands were labelled using keyword-based indicators inspired by prior research and converted into fixed-length numerical sequences for processing. The resulting model attained excellent predictive scores, with 99.02% accuracy, 96.70% precision, 99.69% recall, and 98.17% F1-score, which is much higher compared to the more complex transformer-based approaches reported in related work [1]. The trained model was deployed in a Linux VM and used to live-monitor history file which was altered for more live command updates and effectively detect malicious behavior in three realistic conditions: data exfiltration, privilege escalation, and reverse-shell attempts. The system operates fully in user space and imposes minimal overhead, making it suitable for host-level security monitoring. Overall, the results indicate that a lightweight TextCNN model can provide highly accurate, real-time command threat detection and provide a practical foundation for future enhancements involving sequence-aware models and extended log incorporation.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Hafeez, Khadija UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > QA Mathematics > Computer software > Computer Security T Technology > T Technology (General) > Information Technology > Computer software > Computer Security
Divisions:	School of Computing > Master of Science in Cyber Security
Depositing User:	Ciara O'Brien
Date Deposited:	04 Sep 2026 09:32
Last Modified:	04 Sep 2026 09:32
URI:	https://norma.ncirl.ie/id/eprint/9822

Actions (login required)

View Item