Tools
Tools
Vargas Bocanegra, Erik Raúl (2024) Evaluating the Effectiveness of OpenAI a Dedicated Penetration Testing Chatbot in a Comparative Analysis of AI-Assisted and Manual Workflows. Masters thesis, Dublin, National College of Ireland.
Preview |
PDF (Master of Science)
Download (1MB) | Preview |
Preview |
PDF (Configuration Manual)
Download (1MB) | Preview |
Abstract
Penetration testing, a fundamental cybersecurity practice, traditionally involves manual methods that require skilled professionals to identify and exploit system weaknesses. While effective, these manual approaches can be time-consuming. Recent advancements in Large Language Models, such as the OpenAI GPT series, offer a promising hybrid solution that combines automation efficiency with human precision. This study explores the integration of LLM-powered chatbots into penetration testing workflows, focusing on their effectiveness, efficiency, and usability. Through a comparative analysis of manual and chatbot-assisted workflows on retired Hack The Box (HTB) virtual machines, this research measures detection accuracy, false positive rates, task completion time, and exploitation success rates. Chatbot-assisted workflows exhibited higher detection accuracy (93% vs. 85%), lower false positive rates (9% vs. 14%), and significantly faster task completion times (28% reduction). Qualitative feedback highlighted the chatbot’s adaptability and iterative guidance, although limitations in handling novel vulnerabilities and domain-specific questions were observed. The findings suggest that LLM-based tools can significantly enhance penetration testing, especially for routine and structured tasks. However, human expertise remains essential for complex, non-standard scenarios. This research underscores the transformative potential of LLMs in advancing cybersecurity practices.
Actions (login required)
 |
View Item |