Bidirectional LSTM approach to image captioning with scene features

Agughalam, Davis Munachimso, Pathak, Pramod and Stynes, Paul (2021) Bidirectional LSTM approach to image captioning with scene features. In: Proceedings of SPIE: The Thirteenth International Conference on Digital Image Processing (ICDIP 2021). SPIE, 118780B. ISBN 9781510646018

Full text not available from this repository.

Abstract

Image captioning involves generating a sentence that describes an image. More recently, it has been driven by encoderdecoder approaches where the encoder such as convolutional neural network (CNN) can extract the visual features of an image. The extracted visual features are passed to a decoder such as a long short-term memory (LSTM) network in order to generate a sentence that describes the image. One major challenge with this approach is to precisely include the scene of an image in the generated sentences. To resolve this challenge, visual scene features have been used with unidirectional LSTM decoders. However, for long sentences, this limits the precision of the generated text. This research proposes a novel approach to generate sentences using visual scene information with a bidirectional LSTM decoder. The encoder is based on Inception v3 to extract the object features and Places365 to extract the scene features. The decoder uses a bidirectional LSTM to generate a sentence. The encoder-decoder model is trained using the Flickr8k dataset. Results show improved performance for generating longer sentences with a 9% increase in BLEU-3 and a 12% increase in BLEU-4 scores compared to compared to other encoder-decoder methods that are limited to only using global image features. Visually impaired people that use screen readers would benefit from this research as they would get an enhanced description of an image that includes the background scene thereby creating a wholesome picture in the mind of the reader.

Item Type:	Book Section
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science Q Science > QA Mathematics > Computer software T Technology > T Technology (General) > Information Technology > Computer software
Divisions:	School of Computing > Staff Research and Publications
Depositing User:	Clara Chan
Date Deposited:	28 Jul 2021 12:19
Last Modified:	15 Mar 2022 11:57
URI:	https://norma.ncirl.ie/id/eprint/4906

Actions (login required)

View Item