This study investigated the potential of utilizing ExamSoft scores and exam-taking snapshot behavioral data to assess the impact of self-explanation prompting (Chi et al., 1994) on pharmacy students' academic performance and test-taking strategies. Given that multiple-choice examinations are critical in the pharmacist licensure process, understanding effective test-taking strategies is essential for future pharmacists’ success.
This study was conducted in the context of a third-year (P3) pharmacotherapy course, in which students are expected to apply advanced clinical reasoning skills. By examining the impact of self-explanation prompting on exam performance and test-taking behaviors, this research aims to inform the development of optimal assessment strategies and associated instructional methods to help pharmacy students build strong self-explanation skills.
Doctor of Pharmacy (PharmD) students at a Midwestern university use ExamSoft software for all summative assessments. ExamSoft generates both detailed performance data and individual exam-taking logs called snapshots. While originally designed to deter dishonesty, ExamSoft snapshot logs can provide valuable insights into students’ test-taking strategies (Cernusca, 2021). The Renal, Fluids & Electrolytes Pharmacotherapy course has consistently presented challenges, with both high- and low-achieving students underperforming on certain assessment items. The recurring need to adjust scores for specific outlier assessment items raised concerns about whether these performance issues stemmed from knowledge gaps or ineffective exam-taking strategies.
Self-explanation, identified by Chi et al. (1989) as a key cognitive strategy among high-achieving students, involves leveraging prior knowledge to construct meaningful explanations. Self-explanation is assumed to work because it persuades learners to fill gaps in their own knowledge, helps them adapt procedures from an initial context to a more general one, and, in turn, facilitates analogical problem-solving (VanLehn et al., 1992). While naturally occurring in proficient problem solvers, self-explanation can be stimulated in all students through guidance or prompting (Bielaczyc et al., 1995; Chi et al., 1994). Researchers focused on a range of prompting categories, such as verbal versus written prompts, planned or just-in-time inquiry prompts, and prompts provided by the instructor or during a peer review session.
To explore its impact, just-in-time written self-explanation prompting was introduced for the most challenging items on the chronic kidney disease exam to assess its effects on both performance and exam-taking behaviors. The target assessment items used in the exam used a case-based multiple-choice format, and the self-explanation prompt was introduced at the end of the stem, as exemplified below.
GD is a 75-year-old male with Stage 4 kidney disease. He presents to the nephrology clinic with fatigue, shortness of breath, and dizziness. He denies any bleeding and does not associate his symptoms with increased activity or exertion. Labs were drawn as follows: SCr: xxx; Albumin: yyy; Serum zzz; Vit D:vvv (deficient) PO4-: yyy. In regards to MD's phosphate, what intervention do you plan to initiate when MD leaves the clinic today?
[Self-explanation prompt] When evaluating this question, think back to what laboratory values are needed to initiate treatment and the overview of treatment.
A…/B…/C…/D…
Research design, sampling, and data analysis
This study used a sequential mixed-methods design, integrating a quasi-experimental quantitative phase followed by a qualitative analysis phase to examine the impact of self-explanation prompting on pharmacy students' exam performance and test-taking behaviors (see Figure 1).
Figure 1
Sequential Mixed-Methods Design
A total of 48 students participated in the exam. Their de-identified data available in ExamSoft was retrieved after the final scores were officially recorded for use in this study. The quasi-experimental phase involved comparing performance on difficult multiple-choice questions with control items lacking self-explanation prompts and paired intervention items incorporating them. A paired t-test was used to compare the overall performance of the control and intervention groups. In addition, the exam scores of the top 10 and bottom 10 performers were compared using a one-way ANOVA.
In the qualitative phase, ExamSoft snapshot action logs for the top 10 and bottom 10 performers were retrieved and downloaded as Excel spreadsheets. These snapshot logs include all actions during the exam for each assessment item, along with the timestamp for each action. The information from the snapshot logs was then organized by assessment item and time stamp for the analysis phase (see Figure 2). The reorganized snapshot data were then qualitatively analyzed to identify beneficial and damaging exam-taking behavioral patterns. We identified two beneficial behavioral patterns, (b1) - initial correct answer maintained to the end, and (b2) - wrong answer in the sequence corrected by the end, and two damaging behavioral patterns, (d1) - change of a correct answer to a wrong one, and (d2) - selecting and maintaining the wrong answer to the end. These behavioral traits potentially reflect students’ engagement in building self-explanation skills.
The behavioral patterns were then quantified and compared between control and intervention assessment items using both a paired-samples t-test and a one-way ANOVA to assess students’ engagement in self-explanation tasks in the two quasi-experimental conditions.
Figure 2
Sample Snapshot Log and Associated Analysis Spreadsheet
A paired-samples t-test revealed that overall, students performed significantly better on intervention questions with self-explanation prompts (M = 82.3%, SD = 28.2) than on control questions (M = 34.4%, SD = 32.9), t(47) = -7.8, p < .001 (see Figure 3a). A one-way ANOVA comparing the top and bottom 10 performers confirmed significant performance gains in both groups when self-explanation prompts were used (see Figure 3b).
Figure 3
Performance Score Results: (a) Overall Score; (b) Top and Bottom 10 Performers.
Additionally, as shown in Figure 3b, the initial 25% performance gap between high and low achievers on control items was reduced to 15% for the intervention items. However, this difference was not statistically significant (p = .055).
The results from the analysis of the top and bottom 10 performers provided behavioral support for the performance findings. Overall, for both pairs of questions used in this study, a paired-samples t-test indicated that beneficial behaviors significantly increased from 38% on control questions to 83% on intervention questions, t(73.8) = -4.6, p < 0.001. For both bottom and top performers, a one-way ANOVA showed that the combined beneficial behavioral traits in the intervention groups were significantly higher than in the control group (see Figure 4).
A more granular analysis of specific behavioral traits showed that, while the bottom and top 10 performers started at different levels, both groups increased the number of assessment items they got correct from the beginning to the end, with 80% (see Figure 5a). On the other hand, looking at the damaging behavioral traits, the 10 bottom performers reduced from 90% to zero the number of questions that they got wrong from the beginning to the end, while the top 10 performers decreased the same damaging behavioral trait from 50% to zero (see Figure 5b).
Figure 4
Overall Beneficial Behavioral Traits for the 10 Top and Bottom Performers
Figure 5
Changes in Specific Behavioral Traits by Performance and Intervention Group. (a) Beneficial Behavioral Traits, (b) Damaging Behavioral Traits
 |  |
(a) | (b) |
This study confirmed that self-explanation prompts enhance pharmacy students' exam performance, consistent with previous research (Ionas et al., 2012). Analysis of ExamSoft snapshot logs showed that self-explanation prompting fostered beneficial test-taking behaviors such as retaining correct answers or revising incorrect ones. This indicates its role in stimulating students’ engagement in self-explanation during the exam when a self-explanation prompt was used.
We can therefore infer that while most pharmacy students possess the required prior knowledge to perform well on multiple-choice exams, they may lack effective exam-taking strategies. On one hand, these findings highlight the untapped potential of the rich datasets generated by examination software for deploying and monitoring student-centered assessment and instructional strategies. On the other hand, because the self-explanation prompting intervention led to beneficial changes in students’ exam-taking behaviors, future research will need to explore the development and implementation of a self-explanation prompting scaffolding strategy.
Because the focal course is team-taught, future research will focus on strengthening students’ self-explanation behaviors through a series of interventions that use self-explanation prompts with varying levels of fading across two similar courses taught by the same instructor (see Figure 6). Finally, we intend to analyze the potential to develop specialized AI tutors that will guide students in solving case-based multiple-choice questions using fading self-explanation prompts.
Figure 6
Future Research Structure
EdTech Archives