Date
Winter 12-8-2025
Advisor
Dr. Matt Brown
Program Director
Dr. Jamie Stacy
Document Type
Paper
Abstract
This study compares the performance of knowledge-based expert systems (KBES) and large language models (LLMs) in narrow-domain tasks. Using Akinator as the representative KBES and ChatGPT as the representative LLM, fifty character-identification trials were conducted. Results show that both systems ultimately succeeded in identifying all characters, but their efficiency and accuracy differ. Akinator required fewer incorrect guesses and produced no identifiable total failures, or “errors,” while ChatGPT occasionally erred beyond possible continuation despite similar average guess counts. Statistical analysis revealed no significant difference in the number of questions required before success, but McNemar’s test indicated that ChatGPT made significantly more incorrect guesses. These findings suggest that, while LLMs can rival KBES in efficiency, expert systems retain an advantage in accuracy within specialized domains.
Recommended Citation
Lange, Carter A., "An Assessment and Comparison of Expert System Performance and Large Language Model Performance" (2025). ATU Honors Projects. 3.
https://orc.library.atu.edu/atu_honors/3
Included in
Applied Statistics Commons, Business Analytics Commons, Categorical Data Analysis Commons