Analyzing Middle School Students’ Performance in a Multimedia-Enriched Problem-Based Learning Program Through Learning Analytics

Problem-Based Learning (PBL) is a student-centered approach where students learn by solving problems and reflecting on their experiences (Barrows & Tamblyn, 1980). While research has shown the benefits of PBL for K-12 students, it also reveals challenges teachers face in implementation. Recent studies suggest learning analytics (LA) can provide key insights into students' learning processes in open-ended environments like PBL, helping teachers identify when scaffolding is needed. However, research on LA use in K-12 settings is still limited. This study uses LA to examine middle school students' performance in a PBL context, exploring how students' content knowledge relates to learning performance and behaviors.

418 sixth grade students, comprising 211 boys and 207 girls, from three different schools located in the western and southwestern of the U.S, used a 3D immersive PBL program. Through this program, they learned space science by rescuing six different alien species and relocating them to suitable habitats. This program has nine built-in multimedia tools to facilitate students’ problem-solving with instructional scaffolding (Table 1). These tools enable students to engage in cognitive activities to solve problems and are designed to support students’ scientific inquiry (Liu et al., 2021).

This program incorporates problem-solving tasks, which involve researching various aliens and planets, designing and launching probes to the planets they think are most suitable for each alien, and validating their hypotheses by analyzing data collected from the probes. Students are asked to provide a rationale for why they plan to launch a probe and their probe justifications were categorized into five types (i.e., random input, vague inquiry, vague claim, specific inquiry, and reasoning). To execute these problem-solving tasks, students are required to strategize, collect relevant information, and decide on using the embedded tools effectively.

In this study, a 20-item multiple-choice science knowledge test was used to measure students’ understanding of key science concepts introduced in the PBL program (Cronbach’s alpha =.84). The tests were given before and after students used the PBL program. In addition, over 30,000 lines of log data on students’ just-in-time trajectory of students’ navigation patterns were captured to understand how each student used the tools to solve the problem within the PBL program. The log data were cleaned and used to calculate the frequency (how many times each tool was accessed) and duration (how long a student used each tool in minutes) of each student’s tool use. The students’ learning behavioral patterns were measured with the frequency and duration of using each tool and the counts of sending probes.

Table 1

Technology-Enriched Tools in the Problem-Based Learning Program

Relationship between Students’ Science Knowledge and Their Behavioral Patterns

All students were categorized into low, medium, and high knowledge groups, given their science knowledge test scores. Kruskal–Wallis tests investigating differences among science knowledge groups based on their post-science test scores revealed significant differences in frequency and duration of tool usage in Concept Database, Spectra, and Notebook, as well as in duration for Periodic Table and Alien Information Center (Table 2). Post hoc Mann–Whitney U tests detected significant differences between high and low knowledge groups in these aspects (Table 3). Between the high and medium knowledge groups, significant differences were found in the durations of Periodic table and Alien Information Center. For the medium and low knowledge groups, significant differences were found in the frequencies of Concept Database and Notebook.

Students in the low science knowledge group spent significantly more time with those tools addressed above and used them more frequently in problem-solving than the high science knowledge group did. Since these tools are designed to facilitate students’ science learning and to help them solve problems, the findings suggest that the students with low science knowledge needed more support and content scaffolding to solve the problem.

Table 2

Findings from Kruskal-Wallis Tests Examining Overall Behavioral Patterns Grouped by Post-Science Knowledge Test Scores

*p < .05.

Table 3

Findings from Mann-Whitney U Tests Between Students Grouped by High, Medium, and Low Groups of Post-PBL Science Knowledge Scores

*p < .05, **p < .02, ***p < .01.

Relationship Between the Changes in Students’ Science Knowledge and Their Behavioral Patterns

We explored the behavioral patterns of the groups in relation to changes in their science knowledge from pre- to post-PBL. Students were categorized into three groups: (a) low-high (n = 32), those students whose science knowledge improved from low on the pre-PBL test to high post-PBL; (b) low-low (n = 29), those whose science knowledge was low on both tests; and (c) high-high (n = 14), those whose science knowledge was high on both tests. Kruskal-Wallis Tests revealed significant differences among these three groups of students in tool usage frequencies and durations for the Solar System Database, Notebook, Probe Design Center, Mission Control Center, and Concept Database (Table 4). In addition, the total number of probes launched, the count of successfully launched probes, and the proportion of probe justification types were significantly different among the three groups. Post hoc Mann–Whitney U tests highlighted significant differences between the low-low and high-high science knowledge groups in tool usage frequencies and duration for the Solar System Database, Notebook, Probe Design Center, and Mission Control Center, as well as the frequency of Probe Design Center (Table 5). Furthermore, significant differences were found between the low-high and high-high groups for all variables except the frequency of using the Solar System Database. The high-high science knowledge group used some tools (e.g., Solar System Database) for a shorter period and less frequently but spent more time and used the probe design-related tools more often than the others. As a result, they launched greater numbers of probes and had more successful probes than the others. This finding highlighted the role of probe design in the problem-solving process.

Table 4

Findings of Kruskal-Wallis Tests Examining Overall Behavioral Patterns Grouped by Their Pre- and Post-PBL Science Knowledge Test Scores

*p<.05, **p<.02, ***p<.01

Table 5

Findings from Mann-Whitney U Tests Between the Students as Grouped by Low-Low, Low-High, and High-High Levels of Pre- & Post-PBL Science Knowledge Scores 

*p < .05, **p < .02, ***p < .01

Implications

The findings have several implications for teachers’ PBL implementation. First of all, teachers may need to make extra efforts with students who lack sufficient content knowledge to solve problems, and teachers should monitor their progress more frequently to provide support (Kim et al., 2019). Providing teachers with adaptive dashboard systems which show students’ progress during PBL can be helpful (Tissenbaum & Slotta, 2019; van Leeuwen et al., 2022). Secondly, the high-high group’s frequency and duration of probe design-related tool uses (i.e., the Mission Control Center and Probe Design Center) suggest the importance of testing hypotheses in problem-solving, which aligns with the findings of Spires et al. (2011) regarding the positive relationship between middle-grade students’ hypothesis-testing strategies and their learning outcomes in PBL. Students who entered the program with limited science knowledge often dedicated significant time to using tools designed to teach them the necessary scientific concepts. This reduced the amount of time available for completing problem-solving tasks. In contrast, students who already possessed a strong foundation in science prior to the PBL experience were able to allocate more time and effort toward hypothesis testing, as they did not need to spend as much time on the initial stages of problem-solving.

The study enriches existing literature in two aspects. First, by conducting in-depth analyses of students’ behaviors via LA, we have shown that students’ knowledge of the content plays a role in how students determine when and what tools to use to facilitate their problem-solving. Second, this study offers evidence and insights for researchers in PBL and LA who are interested in designing technology-enriched tools to support K-12 teachers in classroom implementation.

One of the limitations of this study is its reliance on log data as the sole source for analyzing student behavior. Future research could incorporate multiple data sources to enhance the transparency and robustness of data interpretation. Additionally, the study’s focus on a single PBL program limits the generalizability of the findings.

References

1. Barrows, H. S., & Tamblyn, R. M. (1980). Problem-based learning: An approach to medical education. Springer Publishing Company.
2. Kim, N. J., Belland, B. R., & Axelrod, D. (2019). Scaffolding for optimal challenge in K–12 problem-based learning. Interdisciplinary Journal of Problem-Based Learning, 13(1), 3. https://doi.org/10.7771/1541-5015.1712 
3. Liu, M., Pan, Z., Cai, Y., Shao, P., & Han, S. (2021). The effect of a multimedia-enriched problem-based learning environment on socioeconomically disadvantaged middle school students’ science learning: Examining the relationship among self-efficacy, attitude, and performance. Journal of Educational Multimedia and Hypermedia, 30(4), 359-391.
4. Spires, H. A., Rowe, J. P., Mott, B. W., & Lester, J. C. (2011). Problem solving and game-based learning: Effects of middle grade students’ hypothesis testing strategies on learning outcomes. Journal of Educational Computing Research, 44(4), 453–472. https://doi.org/10.2190/EC.44.4.e    
5. Tissenbaum, M., & Slotta, J. (2019). Supporting classroom orchestration with real-time feedback: A role for teacher dashboards and real-time agents. International Journal of Computer-Supported Collaborative Learning, 14(3), 325–351. https://doi.org/10.1007/s11412-019-09306-1 
6. van Leeuwen, A., Teasley, S. D., & Wise, A. F. (2022). Teacher and student facing learning analytics. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), Handbook of learning analytics (pp. 130–140). Society for Learning Analytics Research.