Unique Presentation Identifier:
88
Program Type
Honors
Faculty Advisor
Dr. Matt Brown
Document Type
Presentation
Loading...
Location
Online
Start Date
9-4-2026 8:00 AM
Abstract
This study compares the performance of knowledge-based expert systems (KBES) and large language models (LLMs) in narrow-domain tasks. Using Akinator as the representative KBES and ChatGPT as the representative LLM, fifty character-identification trials were conducted. Results show that both systems ultimately succeeded in identifying all characters, but their efficiency and accuracy differ. Akinator required fewer incorrect guesses and produced no identifiable total failures, or “errors,” while ChatGPT occasionally erred beyond possible continuation despite similar average guess counts. Statistical analysis revealed no significant difference in the number of questions required before success, but McNemar’s test indicated that ChatGPT made significantly more incorrect guesses. These findings suggest that, while LLMs can rival KBES in efficiency, expert systems retain an advantage in accuracy within specialized domains.
Recommended Citation
Lange, Carter A., "An Assessment and Comparison of Expert System Performance and Large Language Model Performance" (2026). ATU Scholars Symposium. 10.
https://orc.library.atu.edu/atu_rs/2026/2026/10
An Assessment and Comparison of Expert System Performance and Large Language Model Performance
Online
This study compares the performance of knowledge-based expert systems (KBES) and large language models (LLMs) in narrow-domain tasks. Using Akinator as the representative KBES and ChatGPT as the representative LLM, fifty character-identification trials were conducted. Results show that both systems ultimately succeeded in identifying all characters, but their efficiency and accuracy differ. Akinator required fewer incorrect guesses and produced no identifiable total failures, or “errors,” while ChatGPT occasionally erred beyond possible continuation despite similar average guess counts. Statistical analysis revealed no significant difference in the number of questions required before success, but McNemar’s test indicated that ChatGPT made significantly more incorrect guesses. These findings suggest that, while LLMs can rival KBES in efficiency, expert systems retain an advantage in accuracy within specialized domains.