A RAG Framework for Legal Text Retrieval and Generation in Irish Statutory Instruments

Hassan, Ozair (2025) A RAG Framework for Legal Text Retrieval and Generation in Irish Statutory Instruments. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (1MB) | Preview
Preview	PDF (Configuration Manual) Download (825kB) | Preview

Abstract

Retrieval Augmented Generation (RAG) systems are the backbone of grounding responses from large language models in external knowledge, but ensuring accuracy and efficiency in certain specialized domains remains an ongoing challenge. This research proposes a comparative framework for evaluating the three main RAG approaches those being BM25, vector based and hybrid (BM25 + vector). A custom dataset was prepared from the Irish Statutory Instruments (2020-2025) by extracting the text from the PDFs, applying section-based chunking and encoding the chunks using BAAI/bge-large-en-v1.5. A gold standard evaluation set of 240 question-answer pairs was used for benchmarking. Weaviate database was utilized in the implementation of BM25, vector and hybrid retrievals. Evaluation was primarily focused on the following metrics, Mean Average Precision, recall, Average Entailment Score, Average Contradiction Score, Average Obligation Coverage Score and Average Final Composite Score. Results indicate that the Weaviate based hybrid system with re ranking had a performance increase in retrieval precision of more than 20% over the baseline simple hybrid system. Overall hybrid re rank performed best with 0.702 recall and highest final composite score of 0.35. This research aims to contribute to the practical guidelines for the designing and development of reliable and efficient RAG systems in the legal domain benefitting legal practitioners and policy makers with trustworthy AI tools.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Fajemisin, Ade UNSPECIFIED Stynes, Paul UNSPECIFIED
Uncontrolled Keywords:	Retrieval Augmented Generation; BM25; Vector; Legal document retrieval
Subjects:	K Law > K Law (General) Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence
Divisions:	School of Computing > Master of Science in Artificial Intelligence
Depositing User:	Ciara O'Brien
Date Deposited:	02 Sep 2026 09:32
Last Modified:	02 Sep 2026 09:32
URI:	https://norma.ncirl.ie/id/eprint/9758

Actions (login required)

View Item