Embedding Model Evaluation in Legal RAG Systems for Automobile Law Question Answering

Yelala, Nithin Reddy (2025) Embedding Model Evaluation in Legal RAG Systems for Automobile Law Question Answering. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (652kB) | Preview
Preview	PDF (Configuration Manual) Download (1MB) | Preview

Abstract

The exponential growth of complex legal documents in India has made it difficult for non-experts to access and interpret statutory information which is mainly in domains such as automobile law. This study addresses that challenge by developing a Retrieval-Augmented Generation (RAG) framework for legal question answering by focusing on the Motor Vehicles Act, 1988. The study aims to identify the most powerful embedding model for accurate semantic retrieval and contextual response generation. There are eight embedding models which were evaluated using metrics such as Recall, Precision, nDCG, F1, ROUGE-L, BLEU, and Cosine Similarity. All experiments in this study were conducted exclusively on English legal text. The dataset was preprocessed, chunked, and vectorized, with embeddings stored in ChromaDB and used via LangChain for retrieval and LLaMA-based LLMs for answer generation. This study results showed that multilingual-e5-large achieved the highest semantic similarity (0.9558) and balanced accuracy across all metrics, outperforming both general-purpose and domain-specific models. The findings theoretically advance the understanding of embedding alignment in legal semantic retrieval and, in practice, offer a scalable framework for automated legal assistance that enhances accessibility and transparency in the justice system.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Haycock, Barry UNSPECIFIED
Uncontrolled Keywords:	Retrieval-Augmented Generation; Embedding Models; Legal Question Answering; Semantic Retrieval
Subjects:	K Law > K Law (General) Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science P Language and Literature > P Philology. Linguistics > Computational linguistics. Natural language processing
Divisions:	School of Computing > Master of Science in Data Analytics
Depositing User:	Ciara O'Brien
Date Deposited:	09 Sep 2026 11:24
Last Modified:	09 Sep 2026 11:24
URI:	https://norma.ncirl.ie/id/eprint/9930

Actions (login required)

View Item