Integrating Autoencoders and GPU Acceleration for Proactive Anomaly Detection in ETL Workflows

Yedla, Kranthi Kumar (2025) Integrating Autoencoders and GPU Acceleration for Proactive Anomaly Detection in ETL Workflows. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (757kB) | Preview
Preview	PDF (Configuration Manual) Download (362kB) | Preview

Abstract

This thesis addresses the problem of improving data quality validation within ETL pipelines where the legacy manual methods of data validation tend to be inefficient in identifying minor anomalies and leads to defect leakage into production. To address this we developed a GPU-accelerated anomaly detection architecture, combining autoencoder neural networks, as part of a Dagster-managed ETL pipeline. The method was experimented on three datasets, namely, banking, insurance and sales, with injected anomalies and standard validation checks involving nulls, duplicates and missing records. In a series of experiments, it was shown that the system was highly detective with all injected anomalies being detected with about 1% of the anomalies being detected in banking data and about 0.5 %in insurance data, and in the sales data only low-level rule problems were detected. GPU acceleration was demonstrated to have a significant positive effect on runtime performance, as validation took 12 to 15 seconds to complete versus 20 to 25 seconds on CPU. Altogether, the results indicate that the use of autoencoders with GPUs is capable of improving ETL QA testing because it offers quick, precise, and automated anomaly detection that supplements the conventional data quality control and might lead to leakage of defects into the production.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Hamill, David UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Machine learning
Divisions:	School of Computing > Master of Science in Data Analytics
Depositing User:	Ciara O'Brien
Date Deposited:	24 Aug 2026 12:36
Last Modified:	24 Aug 2026 12:36
URI:	https://norma.ncirl.ie/id/eprint/9608

Actions (login required)

View Item