Date of Award

2-3-2023

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Computer Science

First Advisor

Mariofanna Milanova

Abstract

Retrieving visual or textual similarities from an image query and vice versa has drawn much interest in computer vision. With the growth of the E-commerce marketplace, image retrieval provides excellent competitive opportunities for vendors and customers through a robust recommendation system. Feature integration has always been an essential task for multimodal-based image retrieval approaches. However, different existing matching techniques have been used separately for visual and text similarity. Still, researchers are looking for new methods when it comes to multimodal Image Retrieval. In this paper, I study the image retrieval task, where the input query is an image plus text sentence that describes the image. The system starts a query triggered by input image and text while taking the help of the Transformer model, which puts attention on both modalities and combines embedded features through the feature fusion technique. I proposed a feature fusion layer using modified Text Image Residual Gating in my work. I have used two methods to retrieve images based on the features extracted from the fusion layer. First, I trained K Nearest Neighbor (KNN) algorithm on the training data, and later I used test data to find a similar image. Second, I used the clustering technique-Means Clustering and a support vector machine to compute the nearest neighbor points and cluster the center to find a similar image. From the accuracy results, I found that SVM (Support vector Machine) is more effective than the KNN accuracy results, giving an overall accuracy of 92%.

Download

Share

COinS