PsycoPipe-Multimodal Emotion Analysis Framework Using Fine-tuned SER and ASR models with LLM For Psychiatric Conversations

Madhav, Soumya (2025) PsycoPipe-Multimodal Emotion Analysis Framework Using Fine-tuned SER and ASR models with LLM For Psychiatric Conversations. Masters thesis, Dublin, National College of Ireland.

Preview	PDF (Master of Science) Download (2MB) | Preview
Preview	PDF (Configuration Manual) Download (1MB) | Preview

Abstract

Accurate emotion recognition in psychiatric conversations is crucial for developing effective clinical decision support systems. Recognising emotions from speech and text is important for creating an effective emotion analysis system. Common challenges in this domain are the lack of domain relevant psychiatric conversational data and lack of domain adaptive models. This thesis introduces PsycoPipe, a multimodal framework that integrates fine-tuned Speech Emotion Recognition (SER) and lightweight tiny Automatic Speech Recognition (ASR) with Large Language Model (LLM) analysis to address these challenges. The study applies a two-phase approach Phase 1 develops a synthetic data generation pipeline using Gemma-3 for conversation generation and IndexTTS with EARS voices for realistic emotional speech synthesis. This dataset is utilised in the fine-tuning of Wav2Vec2-base for SER and fine-tuning of Whisper-tiny for ASR applying selective layer unfreezing and hyperparameter tuning to capture psychiatric terminology and emotional expressions. Phase 2 evaluates inference on a separate synthetic test set combining ASR+LLM semantic analysis and SER acoustic features through a late fusion LLM based ensemble with confidence weighting and label mapping. Results show that fine-tuned Whisper reduces Word Error Rate (WER) by 62%, Fine-tuned Wav2Vec2 achieves 92.3% accuracy with 92.9% F1-score and the ensemble outperforms standalone models, lowering mean emotion distance to 17.38% from 32.4% for SER alone. These findings validate the use of synthetic data and domain specific fine tuning, proposing a scalable and privacy-conscious framework for clinical conversation analysis leading to real-world applications in mental health support.

Item Type:	Thesis (Masters)
Supervisors:	Name Email Zahoor, Sheresh UNSPECIFIED
Uncontrolled Keywords:	SER; ASR; Fine-tuning; Whisper; Wave2Vec2; WER; Ensemble; LLM
Subjects:	R Medicine > RC Internal medicine > RC0321 Neuroscience. Biological psychiatry. Neuropsychiatry > Neurology. Diseases of the Nervous System. > Psychiatry Q Science > QH Natural history > QH301 Biology > Methods of research. Technique. Experimental biology > Data processing. Bioinformatics > Artificial intelligence Q Science > Q Science (General) > Self-organizing systems. Conscious automata > Artificial intelligence B Philosophy. Psychology. Religion > Psychology > Emotions R Medicine > RA Public aspects of medicine > RA790 Mental Health
Divisions:	School of Computing > Master of Science in Artificial Intelligence
Depositing User:	Ciara O'Brien
Date Deposited:	04 Jul 2026 13:54
Last Modified:	04 Jul 2026 13:54
URI:	https://norma.ncirl.ie/id/eprint/9473

Actions (login required)

View Item