Real-Time Data Processing in Cloud Environments

Deshpande, Atharav Jayant (2025) Real-Time Data Processing in Cloud Environments. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (694kB) | Preview
Preview	PDF (Configuration Manual) Download (395kB) | Preview

Abstract

This project provides a reproducible blueprint for organisations adopting cloud-native real-time analytics pipelines. The proliferation of data generated by modern digital systems necessitates advanced solutions for its timely processing and analysis. This research addresses the challenge of building scalable and resilient real-time data processing pipelines within cloud computing environments. The project focuses on the financial services sector, where low-latency data analysis is critical for market monitoring and decision-making. The core contribution is the design, implementation, and evaluation of a comprehensive, serverless data architecture on Amazon Web Services (AWS). This architecture ingests a simulated stream of financial market data using AWS Kinesis, processes it in real-time with AWS Lambda, archives it into an AWS S3 data lake, and makes it available for ad-hoc analysis through AWS Glue and Athena. Furthermore, the system includes a robust monitoring component, featuring a custom Flask web application that provides a real-time health overview of the pipeline's underlying infrastructure. The evaluation shows that the system is designed for high throughput, low latency, and automatic scaling, validating that a serverless approach offers an economical and operationally viable means to address real-time data processing problems. The study provides a validated framework for organizations hoping to adopt a cloud native approach for advanced analytics on data.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Gupta, Punit UNSPECIFIED
Subjects:	T Technology > T Technology (General) > Information Technology > Cloud computing Q Science > QA Mathematics > Electronic computers. Computer science > Computer Systems > Computers > Electronic data processing T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science > Computer Systems > Computers > Electronic data processing Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Machine learning
Divisions:	School of Computing > Master of Science in Cloud Computing
Depositing User:	Ciara O'Brien
Date Deposited:	20 Mar 2026 15:06
Last Modified:	20 Mar 2026 15:06
URI:	https://norma.ncirl.ie/id/eprint/9208

Actions (login required)

View Item