Date of Award

12-17-2024

Document Type

Thesis

Degree Name

Master of Science (MS)

Department

Computer Science

First Advisor

Nitin Agarwal

Abstract

This research presents a novel methodology for the collection, processing, and analysis of social media data using a microservices-based architecture. The proposed system integrates multiple data streams from various social media platforms, transforming this information into a unified, JSON-based DataObject model for seamless processing and analysis. Unlike monolithic architectures, the microservices approach offers scalability and flexibility, allowing the system to handle the high velocity, variety, and volume of unstructured social media data, including text, images, and videos. By leveraging NoSQL databases like MongoDB, the methodology efficiently manages data in a semi-structured format, supporting real-time analytics such as sentiment analysis, toxicity detection, narrative extraction, and advanced metrics like morality and emotion analysis. Key features include an event-driven architecture for processing DataObjects, Kafka-driven pipelines for distributing tasks, and a dashboarding interface powered by the ELK stack (Elasticsearch, Logstash, Kibana) for visualizing computed metrics. The system’s scalability is achieved using Kubernetes, with Docker for containerization and Redis for caching. Evaluation of the system's performance includes benchmarks for processing complex social computing attributes, highlighting its ability to process high-velocity data in real-time. This research offers a robust and dynamic framework for decision-makers in fast-paced environments, providing actionable insights from social media in a centralized, efficient manner.

Download

Share

COinS