Mutation Analysis in Deep Neural Networks: A study on Test Suite Effectiveness

Narmula, Adarsh (2025) Mutation Analysis in Deep Neural Networks: A study on Test Suite Effectiveness. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (552kB) | Preview
Preview	PDF (Configuration Manual) Download (324kB) | Preview

Abstract

Even subtle failures in Deep Neural Networks (DNNs) now underlie safety-critical applications like autonomous driving and healthcare where failures can be catastrophic. As much as there has been significant improvement in accuracy of DNN, little has been found to be given to the effectiveness of the test suites that are used to validate the models. A method that would provide a promising solution to DNN reliability would be mutation testing, which is a method of intentionally injecting artificial faults (mutants) to test the adequacy of the tests. The work focuses on the question of how the mutation testing metrics may be used to assess the performance of a DNN test suite and potentially enhance its performance, specifically by improving robustness in safety-critical settings. An in-depth comparative study of the most popular mutation testing frameworks, including DeepMutation, DeepMutation++, MuNN, and others, is provided by comparing their mutation operators, evaluation criteria, and empirical outcomes. The paper also defines such important metrics as Mutation Score, Average Error Rate (AER), and K_Score1/2 and evaluates their diagnostic capability in terms of identifying the weaknesses of the test suite. Through integration of results of various frameworks, the study identifies the advantages and drawbacks, as well as feasible applications of mutation testing in safety-critical environments. The findings show that metric-based mutation testing can not only increase fault detection effectiveness, but also offer actionable information to execute specific test suites improvement. This paper ends with conclusions of how mutation testing can be incorporated into the DNN development lifecycle and how it can be used to achieve safer and more trustworthy AI systems in addressing issues related to equivalent mutants, computational cost, and alignment with real-world fault taxonomies.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Prior, Michael UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence Q Science > QA Mathematics > Computer software > Computer Security T Technology > T Technology (General) > Information Technology > Computer software > Computer Security
Divisions:	School of Computing > Master of Science in Cyber Security
Depositing User:	Ciara O'Brien
Date Deposited:	18 Aug 2026 17:09
Last Modified:	18 Aug 2026 17:09
URI:	https://norma.ncirl.ie/id/eprint/9546

Actions (login required)

View Item