Sign Language Detection: A Comparative Study of Deep Learning Models Using RTDETR, YOLOv8, Faster R-CNN

Vudiga, Devaki Naga Venkata Prasanthi (2024) Sign Language Detection: A Comparative Study of Deep Learning Models Using RTDETR, YOLOv8, Faster R-CNN. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (4MB) | Preview
Preview	PDF (Configuration Manual) Download (1MB) | Preview

Abstract

This work, therefore, investigates the application of deep learning architectures on ISL recognition by comparing state-of-the-art Real-Time Detection Transformer (RT-DETR) against established approaches, such as YOLOv8 and Faster R-CNN. The research studied the efficiency of these architectures in terms of accuracy, resource efficiency, and practical deployability through structured hyperparameter optimization across 36 different configurations. In this experiment, there were 4,410 ISL images of 35 classes, while different parameters such as image resolutions of 480x480 and 640x640, batch sizes of 32 and 64, optimizers like SGD, Adam, AdamW, and training duration of 5, 10, and 15 epochs were used. Contrary to expectations, YOLOv8 topped with the best mAP of 0.8237, outperforming the second-best RT-DETR, which had a mAP of 0.8098, and Faster R-CNN, which had a mAP of 0.8012. YOLOv8 showed very stable results with 100% successful configurations, while RTDETR and Faster R-CNN were more sensitive to memory limitations, succeeding in only 83.33% and 66.67% of configurations, respectively. The findings challenge assumptions about transformer-based architectures' superiority, suggesting that simpler, well-optimized architectures may be more effective for ISL recognition tasks. This research provides valuable insights for practical implementation choices in sign language recognition systems and also emphasizes resource efficiency in model selection for real-world applications.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Nagahamulla, Harshani UNSPECIFIED
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science P Language and Literature > P Philology. Linguistics > Semiotics > Language. Linguistic theory > Gesture. Sign language
Divisions:	School of Computing > Master of Science in Data Analytics
Depositing User:	Ciara O'Brien
Date Deposited:	08 Sep 2025 09:04
Last Modified:	08 Sep 2025 09:04
URI:	https://norma.ncirl.ie/id/eprint/8836

Actions (login required)

View Item