Multi-LLM Ensemble Architecture For Cross-Modal Document Understanding

Addanki, Mahesh Kumar (2025) Multi-LLM Ensemble Architecture For Cross-Modal Document Understanding. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (1MB) | Preview
Preview	PDF (Configuration Manual) Download (1MB) | Preview

Abstract

Existing Retrieval-Augmented Generation (RAG) systems suffer from great difficulties in preserving spatial and semantic relationships amongst text, images, and tables in multimodal document processing, causing information fragmentation and lowered accuracy in comprehension. Although past studies have demonstrated single Large Language Models (LLMs) perform particularly best in certain areas GPT-4o reaching 91% vision-task accuracy, Claude 3.5 Sonnet reaching 93.1% in fact-checking, and Gemini 1.5 Pro winning at long-context processing no existing study has elaborately investigated ensemble architectures composed of multi-LLMs for preserving cross-modal relationships in document understanding systems.

This study bridges the essential gap by introducing a novel multi-LLM ensemble architecture that cleverly orchestrates GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro using content-aware model selection algorithms. The system adopts a document-based knowledge graph with typed relationships to maintain cross-modal relationships, alongside hybrid retrieval processes that blend vector similarity with graph traversal for better discovery of contexts.

Extensive testing involving 1500 queries and 65 varied documents across five industry applications demonstrated significant performance outcomes. Statistical significance (p < 0.001) was observed for 74.04% semantic similarity, indicating a 13.1% absolute improvement compared to single-model methods. The preservation of cross-modal relationships achieved an effectiveness rate of 89.9% across spatial, semantic, hierarchical, and sequential dimensions. Enterprise-scale deployment feasibility was confirmed by 0.65-second average response time with complete system reliability.

The research contributions include: a systematic solution for ensemble coordination in multiple-LLM multimodal document understanding, a novel graph-based approach that preserves cross-modal relationships to address fragmentation challenges in state-of-the-art RAG systems, and an industrial-strength system that demonstrates successful applicability in enterprise settings. This research establishes new performance benchmarks and methodological foundations for smart model orchestration in future retrieval-augmented generation systems.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Qayum, Abdul UNSPECIFIED
Subjects:	Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence P Language and Literature > P Philology. Linguistics > Computational linguistics. Natural language processing Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence > Computer vision Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence > Computer vision
Divisions:	School of Computing > Master of Science in Data Analytics
Depositing User:	Ciara O'Brien
Date Deposited:	24 Aug 2026 14:31
Last Modified:	24 Aug 2026 14:31
URI:	https://norma.ncirl.ie/id/eprint/9612

Actions (login required)

View Item