Date of Award

2-8-2024

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Computer Science

First Advisor

Xiaowei Xu

Abstract

Machine Learning is rapidly advancing at an incredible pace due to an increase in computational size, and availability of data. Data is the cornerstone of machine learning algorithms. Information quality has a profound impact on the performance of machine learning systems. Supervised, Unsupervised, and Reinforcement learning are three major ways of performing machine learning. Self-supervised learning, part of unsupervised learning, has made breakthroughs in engineering and research by using Large Language Models (LLM). LLM are a type of neural network, used for language understanding and generation. Recently, LLM has taken a distinct lead by demonstrating state-of-the-art capabilities in natural language processing tasks, moreover extending state-of-the-art performance, even towards computer vision. Transformer architectures have been a catalyst for this development, leading towards more powerful architectures such as BERT, and GPT to exhibit remarkable human-like performances in downstream tasks such as text generation, summarization, named-entity recognition, and question-answering. Due to such remarkable performance, LLM is becoming ubiquitous in software applications impacting all domains such as healthcare, finance, and legal services. There is a considerable gap in evaluating the quality of information from LLM. The need to ensure higher information quality of data from LLM remains imperative, as data is generated and also used as input for training LLM. Evaluation of information quality from LLM might lead to an increase in the trustworthiness and safety of LLM. To fill the considerable gap in the Information quality of text from LLM, we introduce – Kodai, A Framework for data augmentation of large language models in machine learning. Kodai uses a weighted combination of relevance, accuracy, and consistency as information quality dimensions. We demonstrate feasibility by calculating information quality scores between three datasets and their synthetic data from LLM. Moreover, we perform a three-stage evaluation, first, we translate unstructured census data into structured using named-entity recognition, second, we demonstrate machine learning approach outperforms the rule-based approach, third, we evaluate the information quality of extracted entities. We finally extended our framework to evaluate the information quality of the heart attack dataset with synthetic data, further contributing to predicting heart attack among South Asians, proposing robust risk scores, and demonstrating LLM can be used in myocardial infarction. Kodai provides a quantitative approach to evaluate text from LLM, enhancing reliability, and transparency.

Download

Share

COinS