Early Warning System for Distributed File System Failures Using Temporal Log Analysis

Bachupalli Kollan, Harshitha Reddy (2025) Early Warning System for Distributed File System Failures Using Temporal Log Analysis. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (847kB) | Preview
Preview	PDF (Configuration Manual) Download (360kB) | Preview

Abstract

Distributed storage systems generate large volumes of operational logs, and most available analytical tools remain reactive and only detect the failures after they occur. This limitation restricts the ability to intervene in environments such as the Hadoop Distributed File System (HDFS), where block failures can adversely affect system availability and performance. This paper explores the possibility of using temporal trends on the logs of HDFS to predict failures before they become a reality. A temporal forecasting system was developed which converts raw logs into sequence forms and considers two opposite modelling methods: non-sequential baseline model (XGBoost) and sequential deep learning model (LSTM). This methodological design allows for determining whether modelling temporal dependencies offers quantifiable benefits in the early-warning forecasting. The results indicate that the temporal sequencing approach can effectively predict the occurrence of failure-prone behaviour than frequency-based baselines. The paper proves that proactive failure prediction based on the temporal log analysis is feasible and it represents a step toward better reliability management in distributed storage systems.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Anant, Aaloka UNSPECIFIED
Uncontrolled Keywords:	Early-warning systems; temporal forecasting; distributed storage systems; HDFS sequence modelling; LSTM; XGBoost; proactive failure prediction; reliability management
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Machine learning
Divisions:	School of Computing > Master of Science in Data Analytics
Depositing User:	Ciara O'Brien
Date Deposited:	07 Sep 2026 08:58
Last Modified:	07 Sep 2026 08:58
URI:	https://norma.ncirl.ie/id/eprint/9847

Actions (login required)

View Item