David J. Yaron1 and Ryan P. Dwyer2
1 Department of Chemistry, Carnegie Mellon University, Pittsburgh PA 15213; yaron@cmu.edu
2 Department of Chemistry, University of Mount Union, Alliance, OH 44601; dwyerry@mountunion.edu
Abstract.
Keywords: Educational Data Mining, Online Learning
REAL CHEM is an online courseware system that integrates instruction and practice for a two-semester general chemistry sequence. Unlike online homework systems, it replaces the textbook and practice tools with one environment that captures detailed traces of students’ process of learning [Koedinger et al. (2025)]. Early analyses show that students under the same policies often engage in very different ways. This project introduces a visualization tool for examining student pathways and clustering methods for identifying common engagement patterns. Framed as a nested learning engineering cycle, this work supports iterative, data-informed refinement of courseware design and instructional policies within a larger ongoing learning system [Craig et al. (2025)].
REAL CHEM captures detailed logs of every student interaction. Our visualization maps these interactions by listing courseware opportunities vertically (organized by unit and module) and ordering each student’s actions horizontally over time, with weekly boundaries for context. Symbols indicate interaction types, including formative (Learn By Doing, Did I Get This) and summative checkpoints. Hover actions reveal additional detail. Each pathway is also converted into a linear sequence suitable for clustering analyses.
The tool revealed far greater variation in student behavior than expected, challenging design assumptions. Some course policies yielded more uniform engagement, but many instructors were surprised to see students using the courseware in ways that differed from their guidance. Clustering identified pathways ranging from linear progression to adaptive patterns in which students revisited formative practice based on prior performance. Further work is needed to refine these clusters and clarify their relationship to learning outcomes.
These findings illustrate the value of a learning engineering approach in which learner data is used to iteratively evaluate assumptions, refine design decisions, and adapt instructional supports. Recognizing this variation is important for designing supports, adaptive features, and analytics. Course policies can shape behavior, but their influence varies, underscoring the need for clearer guidance and tools that help instructors monitor and respond to engagement patterns.
Figure 1
Student pathway visualizations illustrate clear differences in engagement.
EdTech Archives