Tyree Cowell1, Aaron Wong2, Caitlin Mills3, April Murphy4, Stephen Fancsali5, Rae Bastoni6, Steve Ritter7
1 Carnegie Learning; tcowell@carnegielearning.com
2 University of Minnesota; wonga@umn.edu
3 University of Minnesota; cmills@umn.edu
4 Carnegie Learning; amurphy@carnegielearning.com
5 Carnegie Learning; sfancasli@carnegielearning.com
6 Carnegie Learning; rbastoni@carnegielearning.com
7 Carnegie Learning; sritter@carnegielearning.com
Abstract.
Keywords: online learning, math education, video engagement, eye-tracking
MATHstream is an innovative digital learning tool from Carnegie Learning, a leading provider of K-12 EdTech solutions used by over 50,000 students in the 24/25 school year. MATHstream provides supplemental math instruction that uses videos, interactive questions, and gamification to support math learning. Feedback from students and teachers during site visits to research partner districts suggested that content in some videos was presented too quickly, especially for English second-language learners, and that having a way to slow playback speed would be helpful. Conversely, previous research on the impact of increased video speeds on learning has actually found that increased video speed (1.25x - 1.5x) is optimal for learning, particularly for young adults (Lang et al, 2020; Mo et al, 2022; Murphy et al, 2023; Liu & Jia, 2025). In response to these conflicting insights, our team sought to investigate the impact of different playback speeds on performance and engagement in the context of MATHstream.
While MATHstream’s target audience is primarily middle school and high school students, we implemented a pilot experiment with adults on the online testing platform called Prolific to get preliminary insight into whether varying playback speeds on the platform was a viable option for improving students’ learning experience before making more substantial changes to the MATHstream platform. In the current study, we exposed participants to videos from MATHstream at 3 different speeds (slow, normal, and fast) in order to measure any impact the changes in speed might have on engagement and math performance.
The figure below presents the Learning Engineering Process as laid out by Kessler et. al. (2023). The process involves multiple, iterative phases moving through identifying challenges, creating solutions, implementing solutions, and investigating impact. Recent applications of this process have since identified that nested loops of this process may exist at various points during the cycle (Craig et. al., 2025). In the context of this study, the larger cycle represents MATHstream as a product solution where the challenge is providing quality supplemental math instruction in ever-changing classroom environments. During the investigation phase of that outer loop, we are able to use data we gathered from both the platform and user feedback to identify new, smaller challenges. In this case, the new challenge is that some students desire slower video speeds, while some research suggests that faster video speeds are optimal for learning. In order to address this challenge, we enter into a nested loop of creation, experimentation, and investigation.
Figure 1. Placement of current research in the Learning Engineering Process
The study was conducted with 159 adults who were non-MATHstream users, and who were recruited through the online testing platform Prolific.
Each participant was randomly assigned to one of nine experimental conditions described in Table 1. Participants viewed a short instructional video on Least Common Multiples taught by one of three instructors and presented in one of three playback speeds (0.75x, 1x, 1.25x). Instructional content was consistent across videos, though the exact scripts varied slightly by instructor.
Table 1.
All 9 conditions combining unique instructors and playback speeds.
Instructor A | Instructor B | Instructor C | |
0.75x (slow) speed | Instructor A Slowed Down | Instructor B Slowed Down | Instructor C Slowed Down |
1x (normal) speed | Instructor A Control | Instructor B Control | Instructor C Control |
1.25x (fast) speed | Instructor A Sped Up | Instructor B Sped Up | Instructor C Sped Up |
Building on prior work from the University of Minnesota (Wong et al, 2023; Hutt et al, 2024), our procedure used webcams to capture eye-tracking data across 3 metrics: Area of Interest Proportion (AOIProp) refers to the proportion of gazes on the video, off-screen proportion (OffscreenProp) refers to the proportion of gazes not looking at the screen, and unique proportion (UniqueProp) refers to the proportion of gazes that are on unique locations. During the video, participants were probed for engagement every 60-80s. Participants were shown 4 questions in order to measure Task Unrelated Thought (TUT) (whether they were thinking about the video or something else), Difficulty Disengaging (would it be difficult to disengage from their current thoughts), Valence (how are they feeling on a scale from 1 = very negative to 7 = very positive), and Boredom (how bored they feel on a scale from 1 = very bored to 7 = not at all bored).
After watching the video, participants were asked to complete a short 15-question math quiz on the math content covered in the video (Least Common Multiples). Finally, they were asked to fill out a short survey to gather feedback on four usability metrics: Difficulty (rate the difficulty of the math problems on a scale of 1-5), Understandability (rate how well they could understand the instructor on a scale of 1-5), Likeability (rate how likeable they found the instructor on a scale of 1-5), and Desire for Speed Change (whether they would slow down or speed up the video). We hypothesized that participants in the sped-up conditions would demonstrate higher disengagement and lower performance on related multiple choice questions when compared to participants in the slowed down condition. We also predicted that these effects would be more pronounced for instructor Instructor B, who was reported by users to speak particularly quickly.
In general, participants that watched videos taught by Instructor A demonstrated and reported lower engagement than those who watched videos with Instructor B and Instructor C. However, this may be confounded with video length as videos taught by Instructor A were markedly longer than those taught by Instructor B or Instructor C (average length of 634 seconds compared with 235 and 333 seconds, respectively). Video playback speed did not have a significant effect on most measures suggesting that 1x speed is sufficient for both engagement and performance. Finally, in a methodological sense, this study provides initial evidence that gaze behavior while watching videos can be used to predict self-reported engagement.
As shown in Table 2, there were no significant differences on any eye tracking metric across speed conditions. However, comparing different instructors, we saw that there was a significantly higher proportion of off-screen gazes for Instructor A compared to Instructor B and Instructor C (p = 0.02). Conversely, there was a significantly lower proportion of on-screen gazes for Instructor A compared to Instructor B and Instructor C (p < 0.01, p = 0.03).
Table 2.
Average values and standard deviation for eye-tracking measures across conditions.
Similarly to the eye-tracking measures, there were no statistically significant differences in self-reported engagement measures across different speed conditions. However, we observed a significantly higher proportion of self-reported task-unrelated thought (TUT) for participants who watched Instructor A’s videos compared to videos taught by Instructor B and Instructor C (p = 0.019, p < 0.01) (Table 3). Additionally, users were more likely to report positive feelings towards Instructor C compared to Instructor A (p < 0.01) and were more likely to report feelings of boredom towards Instructor A compared to Instructor C (p < 0.01).
Table 3.
Mean self-reported engagement measures and standard deviation across conditions.
The results from participants’ scores on the 15-question post-quiz on Least Common Multiples and post-video survey responses indicated no significant differences between different playback speeds (Table 4). We noted that at normal playback speed, participants rated Instructor A as less likeable than Instructor B or Instructor C (p < 0.045)
Table 4.
Mean quiz performance and survey feedback with standard deviations across conditions
Finally, we asked participants in each condition if they would prefer the video to be faster, slower, or keep it as is (no change). For instructors Instructor C and Instructor B, most participants were satisfied with the speed to which they were assigned. While we expected that participants would prefer Instructor B’s instruction in the slow condition, surprisingly, we found that they often wanted her to speed up. Finally, for instructor Instructor A, most participants preferred the sped up (1.25x) speed and participants in Instructor A’s normal speed (1x) condition wanted him to speed up.
Figure 2. Results for survey question asking participants if they wanted the video they watched to be sped up, slowed down, or no change.
The most conspicuous limitation of this study is that it was conducted with adults, whereas users of the MATHstream platform are primarily middle school students between the ages of 11-14. Additionally, participants did not watch videos in the larger context of using the MATHstream platform in a classroom environment as part of either core or supplemental instruction. We have since conducted follow-up studies with middle school students who are MATHstream users, with which we compared these results.
This study showed that changing the video playback speeds from 1x to 0.75x or 1.25x did not have significant impacts on user engagement via eye-tracking or self-reported engagement measures, or on math performance. Most differences in engagement were found when comparing measures across different instructors. Specifically, participants who watched videos taught by Instructor A showed lower on-screen gaze behavior, higher reports of boredom, and lower likeability scores. A likely explanation for these results is that videos taught by Instructor A were substantially longer than those taught by Instructor B and Instructor C. This suggests that the length of the instructional video has a larger impact on participant engagement than the speed at which the instructor is speaking.
In most cases across the conditions, participants reported having no desire to change the speed of the video. These results indicated that the existing 1x speed was sufficient for both learning and engagement in this context. We found this result to be somewhat surprising as several studies (Lang et al, 2020; Mo et al, 2022; Murphy et al 2023; Liu & Jia, 2025) have found 1.25x and 1.5x speed to be more optimal for learning than 1x speed. An additional study from this team, conducted directly in the MATHstream platform with middle school users replicating the 3 speed conditions, found no significant differences in student performance or content completion rates (Cowell et al., 2024). These findings provide evidence that user engagement and performance when watching instructional videos may not be as sensitive to playback speed as other work has suggested (Lang et al, 2020; Mo et al, 2022; Murphy et al 2023; Liu & Jia, 2025). Instead, factors such as instructor quality and likeability may be more likely to drive these outcomes. This has implications in designing video-based educational materials, particularly when course lessons involve multiple instructors delivering lesson content.
EdTech Archives