Optimising Serverless Data Ingestion: Evaluating the Impact of AWS Lambda Configuration on Performance and Cost in Amazon S3 Data Lakes

Selvadurai, Abhishek (2025) Optimising Serverless Data Ingestion: Evaluating the Impact of AWS Lambda Configuration on Performance and Cost in Amazon S3 Data Lakes. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (776kB) | Preview
Preview	PDF (Configuration Manual) Download (1MB) | Preview

Abstract

Optimising serverless data ingestion is increasingly critical for modern, cost-aware cloud data platforms that depend on AWS Lambda and Amazon S3 for scalable, event-driven intake. This study conducts a large-scale empirical evaluation of three key Lambda configuration parameters—memory allocation, batch/part size, and reserved concurrency—and assesses their combined effects on performance and cost. Using a 3×3×3 full factorial design, the work evaluates 54 configurations and 27,000 invocations across both event-driven and multipart ingestion modes. Results show that memory allocation is the leading determinant of performance, with up to 70% p95 latency reduction (from 512 MB to 2048 MB). Across configurations, cold-start duration was insignificant, with values ranging between 0.33 and 0.47 s, which implies that the cost of a cold start is dominated by runtime initialization rather than characteristics of the workload. Batch size and part size imposed moderate overhead due to S3 requests being staged in this way; reserved concurrency had only a weak effect, marginally smoothing throughput. The higher memory tiers had higher costs per millisecond but were the most efficient with regard to overall cost due to reduced execution time. Here, a valid configuration decision matrix for tuning ingestion-heavy Lambda pipelines is given, and some interesting future directions for research might include multi-region evaluation, runtime comparisons, and real-world testing of ingestion patterns, along with adaptive optimization strategies.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Makki, Ahmed UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Cloud computing
Divisions:	School of Computing > Master of Science in Cloud Computing
Depositing User:	Ciara O'Brien
Date Deposited:	01 Sep 2026 10:46
Last Modified:	01 Sep 2026 10:46
URI:	https://norma.ncirl.ie/id/eprint/9734

Actions (login required)

View Item