The use of Artificial Intelligence (AI) has become very prominent in higher education contexts in the last five years (Chu et al., 2022), particularly in enabling faculty members to pose questions and provide just-in-time feedback to students in large courses (Dever et al., 2020). This has been a game-changing feature of AI in higher education, as higher education faculty often faced barriers to implementing feedback (Cragg, 2024) and received criticism for the inadequacies of feedback provided to students in large classrooms (Boud & Molloy, 2013). Specifically, there has been a rise in publications related to automated feedback systems, particularly within STEM in higher education (Deeva et al., 2021). Automated intelligent tutoring systems can provide adaptive feedback to assist students during problem solving (Kochmar et al., 2022) and offer guidance on practice items (Yilmaz et al., 2022).
Feedback in higher education has shown positive effects on learners’ self-efficacy (Ozogul et al., 2008), motivation (Dawson et al., 2019), learning outcomes (Ozogul & Sullivan, 2009; Schunk & Ertmer, 2000), and self-regulation (Zimmerman, 2000). In recent years, technology has become a support tool for instructors while providing feedback in large classes, such as clickers (Hunsu et al., 2016) and mobile app response systems (Teo & Chew, 2015). These solutions for feedback fall short in developing students’ deep conceptual understanding and may not push students to achieve high-level cognitive outcomes in STEM courses (Shapiro et al., 2017). This limitation stems from these solutions’ reliance on multiple-choice questions, which do not allow students to construct their own responses and engage in interactive learning (Chi & Wylie, 2014). Emerging AI technologies may present solutions to reduce the issue of reliance on multiple-choice questions for feedback and address the absence of instant feedback in STEM classes (Roll & Wylie, 2016).
One way to improve student learning and engagement is by developing technology that allows students to write their self-explanations and receive feedback. Self-explanation is a cognitive process that learners can actively engage into processing new information, acquiring skills by associating it with prior knowledge, and articulating responses in their own words to express conceptual knowledge (Bisra et al., 2018; Fonseca & Chi, 2011). Self-explanations coupled with personalized and specific feedback are crucial to maximize the effectiveness of this pair (Nakamoto et al., 2023). Self-explanations have been found to be effective for student learning in STEM domains, especially in Computer Science (Sudol-DeLyser, 2015).
ExplainIt is a web-based classroom response system designed to support students’ self-explanations and provide feedback. Specifically, after creating and posing a question (e.g., “why” or “how” questions) using the Instructor App, students view the question and submit their self-explanation responses by using the Student App. With these functionalities in place, ExplainIt is undergoing further development to support automatic assessment of students’ self-explanation responses and provide personalized feedback to students with large language model (LLM)-based natural language processing (NLP) techniques. Preliminary findings for automatically assessing student self-explanations show promising results by achieving high predictive accuracy, overcoming the limitations of manual evaluation in large classes (Carpenter et al., 2024). This highlights the potential of LLMs to efficiently and effectively assess student explanations, toward implementing an AI-enabled feedback system. In this initial implementation of ExplainIt, we did not utilize the LLM/NLP function but focused on the system and student experiences.
Are there any changes in Computer Science (CS) students’ self-efficacy before and after using the ExplainIt classroom response system?
How do CS students perceive their engagement when using the ExplainIt system?
How do CS students perceive the usability of the ExplainIt system?
The case study took place at a public university in the southeastern United States. The researchers investigated the effects of ExplainIt on students’ self-efficacy, engagement, and usability perceptions.
Participants were 32 undergraduate students in a face-to-face Computer Science course. Participants were between 18 and 28 years old. There were 8 (25%) females, 23 (72%) males, and 1 (3%) did not report gender. In terms of ethnicity, 16 (50%) were white, 13 (41%) were Asian, and 3 (9%) were other.
Four instruments were used to collect data: demographic, self-efficacy, user engagement, and system evaluation. The demographic and self-efficacy survey instruments were administered before the ExplainIt implementation. After the course, the self-efficacy, user engagement, and system evaluation instruments were given. Once all the instruments were completed, researchers downloaded and analyzed the data.
The completion rates for the instruments varied. For self-efficacy, 16 students completed both pre and post surveys. For self-efficacy, the mean score was 3.69 in the pre-survey and 3.80 in the post-survey. The question with the highest gain on self-efficacy was: “I am knowledgeable about the concepts covered in the course,” which increased from 3.23 to 3.89. However, there were no significant differences in overall self-efficacy levels between pre- and post-surveys.
Regarding engagement, 18 students completed the post-survey. The average agreement rating for engagement was 3.48. Particularly, three statements received the highest ratings: “ExplainIt was not confusing to use” (4.28); “ExplainIt was not stressful” (4.28); and “Using ExplainIt to learn was an interesting experience” (4.11).
For system evaluation, 17 students completed it. Students rated the system’s ease of use at 3.94 and the ease of using the system to complete tasks at 3.47, where “1” was coded as “very hard” and “5” as “very easy”. The ExplainIt system evaluation yielded an average agreement of 3.85. The highest-rated statements were “time provided to write answers through ExplainIt was generally enough” (4.35) and “ExplainIt system’s complexity was appropriate” (4.29). Students also rated “the system is easy to learn quickly” very high (4.18).
The thematic analysis of the open-ended questions on usability uncovered several themes. Most students found ExplainIt helpful for learning and understanding the CS course concepts. Students also highlighted the simplicity and usability of the system. In terms of disliked aspects, a few students mentioned software bugs, such as a timeout issue. Students suggested additional features for ExplainIt, such as the ability to display a history log of submitted answers, results with graphs, differentiate between graded and non-graded questions, and an overall summary of answered questions.
In terms of students’ self-efficacy, there was a positive change between pre- and post-instruments. This may be due to the self-explanation prompts and responses that helped students learn CS concepts and improve their self-efficacy perceptions. However, since there were no statistically significant differences in the gain scores, these positive gains were minimal. The observed gains may be attributed to the use of the ExplainIt self-explanation system or simply to students learning the course content and thus building confidence over time. Further research is needed to isolate the effects of the ExplainIt system through an experimental study to determine its specific impact on self-efficacy.
Students evaluated the experience with their system positively and found it easy to use and engaging. As higher education faculty often faced barriers in implementing formative feedback (Cragg, 2024), students may have appreciated the newer instructional approach and self-explanation support in their CS courses. This finding is promising, even though students in this study also reported usability issues, such as system timeouts and software bugs. Despite these issues, students remained engaged with the system and rated the system as easy to use. This may be due to the open nature of interaction in CS courses or the fact that CS majors tend to be more lenient toward software bugs and technical issues. Future studies should investigate a more robust version of the system, free of technical issues, and include students from other STEM disciplines.
To effectively integrate ExplainIt into diverse educational settings, instructors should receive training on best practices for designing and implementing self-explanation prompts. Workshops and professional development sessions can help educators craft questions that elicit meaningful self-explanations and support students in interpreting AI-generated feedback effectively. Additionally, ensuring that AI-generated feedback remains equitable and free from biases will be crucial. Finally, large-scale deployment may require integration with various learning management systems and robust server capacity. Addressing these challenges will be essential to enhancing ExplainIt’s impact across diverse learning environments.
This research was supported by funding from the National Science Foundation under Grants DUE-2111473 and DUE-2111216. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
EdTech Archives