Date of Award

2-3-2023

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Computer Science

First Advisor

John Talburt

Abstract

The Data Washing Machine (DWM) is a known and documented open-source Python Jupyter Notebook project that is the foundation for an Unsupervised Data Curation process. The DWM ingests reference data without a prior data cleansing activity and ultimately runs Entity Resolution (ER) on acceptable entity data to cluster duplicate references within the dataset. The DWM currently has 17 modifiable parameters that are used to help tokenize, cleanse, organize, link and cluster like references. With such a large number of parameters, some type of beginning settings as optimal as possible are needed for the DWM process for it to be useful in many Academic and Commercial use cases. To address the DWM need for these initial parameter settings, a Design Science research methodology is used in the creation of a working software artifact called the Parameter Discovery Process (PDP). The artifact addresses the need of parameter settings for 14 of the modifiable DWM parameters. The subset of chosen target parameters is manipulated one at a time using a refactored Java Data Washing Machine used by the PDP to identity reasonable starting parameters. A portion of the parameter settings are decided through regression analysis formulas created from analyzing statistics of known and heavily studied DWM multi sample run outputs. An important new concept of the PDP is the ability to evaluate the parameter changes and to link and cluster like references with and without a Truth File. A truth file is a known true link index for a specific reference dataset. When present, the truth file helps the PDP make parameter change decisions by using the calculated F-Measure for each of the multiple runs of the reference sample file. Even without a truth file, the PDP is able to use data gathered from the run of a DWM parameter change by centering around counts of the iterative linking and clustering processes. Parameter settings decisions are then based on derived values from the gathered data. Reference sample runs that use no truth files produce comparable clustering output very close to the truth file sample runs of the same references.

Download

Share

COinS