Date of Award

11-19-2024

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Information Science

First Advisor

Ahmed Abu-Halimeh

Abstract

In the ever-expanding landscape of digital technologies, the exponential growth of data presents both challenges and opportunities, demanding innovative approaches to data curation. Effective data curation is pivotal for extracting meaningful insights from vast and complex datasets. This study explores the integration of spectral clustering and Shannon Entropy within the Data Washing Machine (DWM), a novel tool designed to streamline unsupervised data curation processes. The DWM incorporates Shannon Entropy into its clustering process, allowing for adaptive refinement of clustering strategies based on entropy levels observed within data clusters. Spectral clustering, known for its ability to handle complex and non-linearly separable data, is investigated as an alternative method to enhance the DWM's capabilities. Additionally, other unsupervised clustering methods, including autoencoders and density-based clustering like DBSCAN, are explored to augment the DWM's ability to handle diverse data scenarios effectively. Rigorous testing of the DWM prototype is conducted on diverse datasets, including various annotated test samples, to assess the performance of spectral clustering in conjunction with Shannon Entropy. The findings reveal promising outcomes, particularly in scenarios with high-quality data. Results indicate that spectral clustering, when combined with entropy-based evaluation, significantly improves clustering outcomes, especially in datasets exhibiting varied density and complexity. However, challenges arise when dealing with poor data quality, emphasizing the importance of data quality assessment and improvement for successful data curation. This study highlights the synergistic role of spectral clustering and Shannon Entropy in advancing unsupervised data curation, offering a more nuanced approach to handling diverse data landscapes. It underscores the practicability of constructing an unsupervised entity resolution engine with the DWM and the necessity of innovative clustering strategies and robust data quality assessments in navigating the complexities of modern data environments.

Download

Share

COinS