Improving processing of real-time Big Data in Smart Grids using Apache Flink and Kafka

Shetty, Supritha (2019) Improving processing of real-time Big Data in Smart Grids using Apache Flink and Kafka. Masters thesis, Dublin, National College of Ireland.

Preview

PDF (Master of Science) Download (2MB) | Preview

Abstract

As we inject more and more "smartness" into energy networks, it unfolds fine-grained data. Around 80 percent of electric meters determined to be replaced by smart meters by end of 2020 in European Union, evidently putting forward plethora of data. Processing and analyzing this Big Data would be complex, time consuming and would have more latency with lesser throughput. Data received by smart meters are at unprecedented speed at real-time, having said that a constant store of data received via smart meters also needs to be addressed, this paper aims to process the data at real-time as well as data received in batches or at store, with higher throughput and lower latency and much faster a execution time. There are different types of existing frameworks and techniques to process Smart grid Big Data being touched upon,this research focuses on using the two most relevant techniques. Apache Kafka being the best combination with Apache Flink using the in-built connector which helps to receive streaming data also being scalabe and fault-tolerant, With that making use of Apache Flink's own features in an optimized manner such as Windowing, its GlobalJobParameters as well as using Flink's Event time and Processing time with Java Programming language. The architecture is used to process Big Data from smart meters to achieve the end result help increasing the overall performance in real-time using cloud based services by deploying the JAR. A significant encouraging results of execution time being lesser when compared to existing approaches can be observed in the results.

Item Type:	Thesis (Masters)
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Cloud computing
Divisions:	School of Computing > Master of Science in Cloud Computing
Depositing User:	Caoimhe Ní Mhaicín
Date Deposited:	23 Mar 2020 17:55
Last Modified:	23 Mar 2020 17:55
URI:	https://norma.ncirl.ie/id/eprint/4145

Actions (login required)

View Item