Comparative analysis of content: human expert vs. artificial intelligence in an exploratory study

Panguluri, Lakshmi (2023) Comparative analysis of content: human expert vs. artificial intelligence in an exploratory study. Masters thesis, Northern Arizona University.

Text Panguluri_2023_comparative_analysis_content_human_expert_vs_artificial.pdf - Published Version Download (1MB)

Abstract

The field of Artificial Intelligence (AI) is currently experiencing a significant transformation characterized by notable advancements in education, healthcare, commerce, banking, agriculture, and transportation. One of the most crucial areas is Generative AI, which represents a fundamental shift in the field. The evolution of large language models within Generative AI is particularly noteworthy, as they demonstrate an impressive ability to generate text that closely resembles human-like language in terms of coherence and relevance. Chat Generative Pre-trained Transformer(ChatGPT), a prominent example of a generative model, has received widespread recognition for its adeptness in understanding and generating text, showcasing the advancements achieved in natural language processing. There has been limited investigation into the ability of ChatGPT to generate text that could be used in places requiring significant human effort. This research aims to initiate an exploratory comparative analysis of the content created by human experts and ChatGPT, employing a set of expert-provided keywords tailored for scripting a meditation activity. As a case study, we generate customized content for the meditation activity within the SUNRISE mobile health application. To ensure a meaningful comparative analysis of the content of a meditation activity, it is crucial to conduct a usability study of the application. This study evaluates the application’s effectiveness and user friendliness, allowing for a comprehensive understanding of its performance. A usability study is employed, utilizing the uMARS (Mobile Application Rating Scale) protocol specifically designed to assess the quality of mobile health applications. This evaluation aims to ensure that the application is user-friendly and meets the necessary standards for effective usability. This research was structured as an empirical exploratory study aimed at understanding user perceptions and experiences. Specifically, the study delved into how individuals with varying levels of familiarity with meditation activities perceived content created by ChatGPT. By strategically employing expert-provided keywords, the research aimed to provide insights into users’ nuanced responses and preferences when presented with content generated by human experts and AI, thereby shedding light on the evolving landscape of text generation by ChatGPT. According to the uMARS findings, the app demonstrates usability with a 3.93 out of 5 rating. Initial results suggest that proficient users(N=6) of meditation rated both human expert-generated and AI-generated content equally. In contrast, novice users(N=6) exhibited a preference for AI-generated content over human expert-generated content. It is critical to note that this exploratory study serves as an initial initiative, and further research is required to delve deeper into the utilization of AI-generated content in healthcare contexts.

Item Type:	Thesis (Masters)
Publisher’s Statement:	© Copyright is held by the author. Digital access to this material is made possible by the Cline Library, Northern Arizona University. Further transmission, reproduction or presentation of protected items is prohibited except with permission of the author.
Keywords:	Artificial Intelligence; ChatGPT; Natural language processing
Subjects:	Q Science > QA Mathematics > QA76 Computer software
NAU Depositing Author Academic Status:	Student
Department/Unit:	Graduate College > Theses and Dissertations College of Engineering, Informatics, and Applied Sciences > School of Informatics, Computing, and Cyber Systems
Date Deposited:	24 Jul 2026 18:36
Last Modified:	24 Jul 2026 18:36
URI:	https://openknowledge.nau.edu/id/eprint/6316

Actions (login required)

IR Staff Record View

Downloads