Date of Award
12-13-2024
Document Type
Dissertation
Degree Name
Doctor of Philosophy (PhD)
Department
Computer Science
First Advisor
Ahmed Abu-Halimeh
Abstract
Entity Resolution (ER) is a critical process in data integration and quality improvement that identifies and links multiple records referring to the same real-world entity. As data volumes and heterogeneity increase, traditional ER methods face new challenges, prompting research into more advanced techniques. The Proof-of-Concept Data Washing Machine (DWM), developed under the NSF DART Data Life Cycle and Curation research theme, aims to automatically detect and correct data quality errors through unsupervised entity resolution. Recent research focuses on enhancing DWM's effectiveness by replacing rule-based methods with machine learning and deep learning approaches, particularly in the linking process. Deep learning models, such as BERT and its variants, are being explored to improve similarity scoring in Cluster ER methods. The integration of these models into the unsupervised DWM process leverages the Attention mechanism to generate reference embeddings and derive similarity score vectors. The research also addresses the optimization of candidate pair reduction in the ER blocking process within the DWM framework. By incorporating machine learning for vectorization and similarity calculation, the goal is to enhance clustering accuracy while maintaining efficiency. A novel approach for handling sensitive data, such as Social Security Numbers (SSNs), is introduced to further streamline pair reduction in the linking stage. After inclusion of ML into DWM, a comparative analysis between Linking_with_ML and SSN_Filtering_with_ML methods across diverse file types reveals that SSN_Filtering_with_ML generally achieves higher precision and a more balanced trade-off between precision and recall. This suggests its robustness and accuracy in the entity matching process, contributing to the overall improvement of the Data Washing Machine's capacity for accurate record linkage while minimizing unnecessary comparisons
Recommended Citation
Sajid, Bushra, "Advanced Models for Linking Process in Data Washing Machine" (2024). Theses and Dissertations. 1241.
https://research.ualr.edu/etd/1241
