Design and Pilot Evaluation of a Gamified Narrative Chatbot for STEM Education

Extended Abstract

Introduction

The rise and ease of access to Generative Artificial Intelligence (GenAI) opens up opportunities for personalized experiences in many spheres, including education. Research has shown that learning can be supported through gamification (Plass et al., 2015) by enhancing motivation and engagement, Socratic instruction (Favero et al., 2024) which fosters critical thinking through guided questioning and by narrative storylines (Gee, 2003) which promote immersion and meaning making in educational contexts. The current presentation showcases the nested learning engineering cycle (Goodall & Kolodner, 2023; Totino & Kessler, 2024), including the design process, a pilot study, and the preliminary results for three gamified chatbots incorporating GenAI and narrative writing to help students learn a STEM topic.

Method and Analysis

Three chatbots incorporating GenAI were designed in an iterative prompt engineering and testing process, utilizing ASU’s CreateAI Builder platform (Ahmed et al., 2025). The goal of these chatbots was to teach participants about Photosynthesis through a socratic method. Two of the three chatbots were gamified (reward-based and role-based) and included a narrative storyline. The third was a baseline chatbot without gamified or narrative features designed for comparing results.

We conducted a pilot study to test these chatbots with 53 undergraduate student participants through ASU’s SONA psychology subject pool. Participants filled out a pre-study quiz about the topic taught by the chatbot, interacted with one of the three chatbots to solve 3-4 challenges, then filled out a post-interaction knowledge quiz, followed by a survey related to the user experience with the chatbot they interacted with.

Pre- and post-test scores were compared for each participant. We also analyzed chatbot interaction logs for bot response quality and participant engagement. The surveys were qualitatively and quantitatively analyzed to test participant learning, engagement, and participants’ prior experience with games, text-based games and GenAI tools.

Findings

Preliminary findings show that participants found the chatbot interactions to be engaging, coherent, and contributing to their knowledge on the topic taught. Despite all chatbot versions contributing to student learning, participants reported enjoying the gamified features offered in both gamified chatbots tested. We will report specific findings about the chatbot features and user experiences with the chatbots and other similar technologies.

Implications. Study findings contribute to the understanding of the novel application of GenAI for gamified learning environments and the interaction of GenAI with socratic learning support. Findings will inform future iterations of the chatbot, with the goal of being employed as a tool in classes, or a supplementary tool for learners of STEM topics.

Acknowledgments

We would like to acknowledge work done by Alison Nguyn in supporting parts of the analysis. This work has been supported by  

References

1. Ahmed, I., Alvarado, P., Jain, S., Arner, T., Reilley, E., & McNamara, D. S. (2025). Arizona State University CreateAI Platform. Grantee Submission.
2. Favero, L., Pérez-Ortiz, J. A., Käser, T., & Oliver, N. (2024, October). Enhancing critical thinking in education by means of a Socratic chatbot. In International Workshop on AI in Education and Educational Research (pp. 17-32). Cham: Springer Nature Switzerland.
3. Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers in entertainment (CIE), 1(1), 20-20. https://doi.org/10.1145/950566.950595 
4. Goodell, J., & Kolodner, J. (Eds.). (2023). Learning engineering toolkit: Evidence-based practices from the learning sciences, instructional design, and beyond. Taylor & Francis.
5. Plass, J. L., Homer, B. D., & Kinzer, C. K. (2015). Foundations of game-based learning. Educational Psychologist, 50(4), 258–283. https://doi.org/10.1080/00461520.2015.1122533 
6.  Totino, L., & Kessler, A. (2024). “Why did we do that?” A Systematic Approach to Tracking Decisions in the Design and Iteration of Learning Experiences. The Journal of Applied Instructional Design, 13(2) https://doi.org/10.59668/1269.15630