Online learning is the education industry’s fastest-growing market (Peck, 2023). Since 2019, computer science has been one of the top three academic fields behind the growth (Online Education Trends Report, 2023). Combining techniques and theory from computer science, statistics, mathematics, and other fields, data science (DS) is described as a relatively new and fast-growing discipline (Meng, 2019; Mike, 2023) and is included in STEM (Science, Technology, Engineering, Mathematics). Probability, the cornerstone of statistics and machine learning (ML), has recently been extremely popular in online courses (Al-Asfour, 2012; Allen & Seaman, 2015).
Achieving high-quality online DS courses that foster student engagement in the online learning environment demands considerable course design efforts, including some needed universally across all areas and others specific to DS courses and statistics.
Research in the ID community suggests that, in any domain of higher education (HE), a successful design of an online learning experience involves the instructor, who is primarily responsible for the subject content (Milosch, 2018), collaborating with an ID (Brigance, 2011) that possesses the following qualities: (a) a solid design foundation supported by learning theories, (b) an understanding of the cognitive process of learning, (c) an ability to utilize research to inform practice, (d) competency in multimedia and online educational format, and commitment to perpetual learning and readiness for the challenge (Hart, 2020).
Through the fostered collaboration, the instructor and the ID prioritize the learning needs of the students. Ultimately, the collaboration analyzes and implements several rounds of design and redesign by collecting students’ feedback and evaluation, which iteratively allows each student to have a unique and satisfying learning experience (Carliner & Chen, 2024). The collaboration throughout the iterative process not only implements a structured modular design in a learning management system (LMS) but also creates continuous student engagement opportunities and activities that foster interacting with learning content and collaborating with peers and constantly incorporates feedback collected from students and other teaching professionals -including teaching assistants (TAs)- into the redesign process. Working with the ID, the instructor creates a cutting-edge, superior course incorporating technology and multimedia into the teaching (McDonald et al., 2022).
With its accumulated research and experience, the instructional design community can help create a common ground for discussion among DS instructors to address challenges specific to their fields, particularly those faced by the statistics component of DS.
There are very few instances of collaboration between ID and teachers in the design of DS FAOC, especially in statistics (Burnham et al., 2022; Sanchez & Wang, 2023; Steinberg, 2010; Yang, 2017). Exceptional examples are a recent discussion of the relationship between usability and cognitive load in statistics education (Tawfik et al., 2023), design-based interactive statistical simulations (Gok & Goldstone, 2022), and design-based statistical literacy (Bessant, 1992).
Teaching and designing online statistics courses have many challenges. For instance, statistics courses typically require hands-on learning and live demonstrations of complex concepts and technologies (Yang, 2017), which might be difficult to replicate online (McQuate, 2020). Minimal resources address the effectiveness of pedagogical strategies in using technology, classroom community and feedback, and student engagement for online statistics courses (Chang & Lee, 2022; Flores et al., 2023). This is aggravated by the lack of a unified definition of online learning in the DS education community (Mills & Raju, 2011; McMurtrie, 2023).
The literature review on online education in DS (focusing on statistics) reveals the fragmented nature of instructors' concern for online learning experiences. Most researchers/instructors study how a specific technological tool helps students learn (Biehler, 2022; Flores et al., 2023). Instructional design literature suffers from the same fragmentation when describing learning experiences in STEM from an instructional design perspective.
Studies of online instruction of DS courses often compare student performance in online and in-person or hybrid courses (George & Leon, 2023; Hoffman & Elmi, 2021; Schwartz et al., 2018; Soesmanto & Bonner, 2019; Summers et al., 2005; Ward, 2004). Some focus on finding differences in student composition through different course modalities (Dutton & Dutton, 2005), and others compare resources to help students of different learning styles learn (Shotwell & Apigian, 2015) or the role of the instructor in online learning or a hybrid course (Tudor, 2006). Few of these approaches focus on the design process or student engagement roles in promoting FAOC quality.
Given all the above, it is necessary to implement and develop effective instructional strategies for online statistics courses (Akdemir, 2010; Yang, 2017) through a collaborative effort between instructors and instructional designers (Milosch, 2018).
To bridge the existing gap between the statistics education and the instructional design literature on what concerns DS online learning experiences, we focus in this paper on the design aspects of a FAOC statistics course, emphasizing the role that instructional design within the UDL and CoI theoretical frameworks and iterative collaboration played in student engagement. The course is an upper-division, calculus-based asynchronous online statistics course, Introduction to Probability.
The iterative design process adopted in our course design collaboration is based on research and evidence. It follows the guidance of the Community of Inquiry (CoI) and Universal Design for Learning (UDL) principles. It aligns with the academic goals of online education, which include, but are not limited to, helping students stay organized, increasing the accessibility of learning, and enhancing the quality of teaching and learning (Abuhassna et al., 2020; Pauldel, 2021).
The iterative design process supports the achievement of academic goals of online education in many ways, including increasing student engagement, collaborative learning, and inclusiveness to guarantee the quality of FAOCs.
A FAOC aims to create a flexible and student-centered learning experience, enables dynamic interaction in the course environment, and encourages and reduces barriers that may be caused by place and time (Alenezi, 2023; Billett et al., 2023; University of Illinois, Springfield, 2024). It also gives access to a broader range of options and resources, such as guest experts, educational technologies, or other materials outside a physical location (Bryson & Andres, 2020; Castro & Tumibay, 2021; Paudel, 2021). Bringing these rich and versatile learning materials to students not only expands the field of knowledge but also strengthens students’ analytical and research skills. In addition, an FAOC encourages creative teaching and seeks excellence in teaching (Mahmood, 2021; Sofi-Karim et al., 2023; University of Illinois, Springfield, 2024), which creates various activities and interactions, such as interactive video lectures, e-Portfolio, blogging, online discussions and more, to prepare students for the next level of learning (Bender, 2023; Fitzgerald, 2022).
As a “systematic process of translating principles of learning and instruction into plans for instructional materials and activities” (Smith & Ragan, 1993, p.2), instructional design plays a crucial role in developing online courses. It should also “be reflective and iterative, in addition to being systematic” (Smith & Ragan, 2004). But iteration requires collaboration and interaction between the instructor and the ID (McDonald et al., 2022; Ustun & Tracey, 2020). Through an iterative instructional design process, the instructor and ID gradually refine the course based on frequent student and ID/instructor evaluations to create engaging activities and use online course functions dynamically within and throughout several course offerings. Data science courses, especially probability, are no exception to the above. The instructor’s subject matter and the designer’s system knowledge collaboratively influence the practice (Tate, 2017). Bond and Dirkin (2020) found that students respond positively to the results of this practice through their increased engagement and satisfaction with the learning experience.
To create a deep, meaningful, and engaging learning experience through interaction and collaboration in this course learning environment, Universal Design for Learning (UDL) principles and Community of Inquiry (CoI) were used to guide the iterative design of the probability course.
Focusing on the course implementation and the learning process (Garrison, 2007; Swan et al., 2012), the CoI framework has been adopted in our online DS course as the framework to iteratively design throughout successive course offerings to improve the learning experience.
The design process creates and supports a constructive, collaborative environment for online learners by developing three interdependent social, cognitive, and instructional presence elements. Social presence is the ability of learners to project their characteristics to the community of inquiry. Cognitive presence is the extent to which the participants in a particular configuration of a community of inquiry can construct meaning through ongoing communication. Teacher presence is planning, promoting, and directing cognitive and social processes to realize personally relevant and educational learning outcomes (CoI, 2021).
The UDL principles are central to the course design process because they guide the creation of a diverse, equitable, and inclusive online learning environment and allow the improvement and optimization of teaching and learning for all people based on a scientific understanding of human learning (CAST, 2024).
The principles of multiple modes of engagement and expression emphasize that instructors and IDs need to consider different activities and methods to encourage student participation and interaction during teaching and learning, bringing in great opportunities for students with diverse learning backgrounds who need to explore the online environment. Using UDL principles as a guide, students with disabilities are supported to reduce barriers to achieving learning goals in both online and hybrid environments using various instructional and support functions (Rao et al., 2021). A study by Katz (2013) that uses UDL principles as an intervention showed significant increases in active engagement, social engagement, and inclusivity. Some recent research applying UDL principles as a basis for online support for mathematics and statistics students during the pandemic also provides strong evidence of the effectiveness of UDL in supporting student engagement and pedagogical and technological integration in the online environment (Lewitzky, 2022).
Introduction to Probability is a calculus-based upper-division course required for statistics and data science majors, minors, and other STEM majors at University of California, Los Angeles (UCLA). This course covers the basic concepts of an introduction to probability course and their application, including sample space and events, probability rules, random variables, univariate and multivariate distributions, expectations, basic probability theorems, and how to use all the above to learn from data. It averages 60-90 enrollments per quarter per offer. The instructor and instructional designer redesigned the asynchronous online version of the Introduction to Probability using the iterative process. They offered it for the Fall, Winter, and Spring quarters from 2021 to 2023. The redesign improved the course quality by implementing reasonable pedagogical strategies to increase student engagement through modular design and technologies, and met the growing demand for this course across campus during and after the pandemic.
According to UCLA’s definition of online asynchronous courses, which is delivering “100% of instructional content to students online” and “all instructional activity (e.g., pre-recorded lectures, online discussion boards), including assessments (e.g., quizzes, exams), is done fully online” (UCLA, 2024, p. 1-2), the fully asynchronous online version of Introduction to Probability structured the course using modular design approach and each learning unit included a coherent learning sequence and rich learning activities, such as instructor presence, student engagement, and appropriate formative and summative assessment opportunities guided by pedagogical strategies from CoI and UDL principles. Each learning unit typically consists of a) an overview and feedback video to summarize previous learning and explain the learning goals that would be achieved through diverse activities, educational technologies, and teaching strategies in the coming week; (b) various presentation formats of learning materials and content, including pre-recorded videos, supplementary materials, articles, infographics, etc.; (c) instructional activities that would be held in the discussion sections with teaching assistants (TAs); (d) assessment and evaluation activities with instant feedback, such as discussion forums, practice quizzes, scaffolded assignments, etc., all aligned with the course learning objectives.
As suggested in the literature reviews, many definitions of instructional design indicate that the course design process should be reflective and iterative in addition to systematic (Smith & Ragan, 2004). Iterative design is an approach where a design is gradually developed and refined based on feedback and evaluation (Cennamo & Kalk, 2019; von Schmieden et al., 2019).
The collaboration between the instructor and the ID iteratively and dynamically includes several key design aspects to improve the course quality and student engagement during the course Introduction to Probability’s initial and subsequent offerings. In the Fall of 2021, the FAOC course debuted with professional studio-filmed video lectures, a well-structured course site on Canvas, a teaching assistant trained by the ID to assist in the FAOC, and significant communication of progress to the ID. The ID was engaged during the first week of the quarter, participating in several activities and helping assess their effectiveness in achieving the engagement goals of the week. Biweekly communication between the ID and the instructor continued throughout the whole course. At the end of the course, the ID and the instructor used the feedback collected from students through canvas surveys at the mid-quarter and end-quarter, students’ performance, and discussions with TAs and instructor to brainstorm and propose the aspects of change, which were all guided by CoI and UDL. This cycle continued in the following quarters: Winter, Spring, and Fall of 2022, and Winter and Spring of 2023. The iterative and collaborative redesign continued what worked well.
The UDL principles suggested implementing multiple modes of engagement, representation, action, and expression to ensure the learning experience for all students (CAST, 2024). One of the collaboration goals was to provide different methods to encourage students to recognize themselves, build a learning community, and interact with the course materials and peers. The methods included but were not limited to integrating multimedia such as filming lecture videos to explain complex concepts and theories so students can watch and review at their own pace, adopting educational technologies such as e-textbooks with easy access, online social networking building tools, and quiz-in-video to encourage more interaction and collaboration among peers, and designing different activities to allow students to apply what had been learned such as case studies and scaffolded assignments. Because the course was offered through Canvas, the design also selected Zoom to offer office hours for students to communicate synchronously with the instructor and TAs, who were also available throughout the learning in discussion sessions to provide students with multiple communication solutions to help with in-class questions and facilitate discussion topics.
Social presence is one of the three components of the CoI framework. Achieving a high social presence in an online learning environment supports the development of personal relationships and encourages collaboration and the creation of a learning community (Aldosari et al., 2022; Singh et al., 2022). Also, Guideline 8.4 in UDL principles emphasizes “foster belonging and community” (CAST, 2024) when designing a learning environment.
After evaluating the social gathering event that usually happened in person in the first week of previous offerings, the instructor and instructional designer re-designed the social event in Gather Town (2023). This virtual space allowed users to interact as if they were in a physical location. By building several virtual spaces in Gather Town, the instructor interacted and shared course expectations easily with students during the two-hour-long social gathering and with a few low-stake activities that had detailed instructions, such as the scavenger hunting syllabus and forming teams based on their conversations, students knew each other quicker and better than in person before collaborating on projects. 95% of students participated synchronously during the social hours and found it was “helpful,” “interesting,” and “enjoyable” (as indicated by students’ survey responses received via a canvas survey).
In addition to the social gathering event, a low-stake structured introduction Canvas discussion activity was added to the beginning of the course that allowed students to present themselves by sharing videos, audio, images, or text. Through this and the Gather Town activity, students gained so much social presence that they wanted to continue with the same groups they formed throughout the course, an essential part of student satisfaction with the social presence aspect. Iterative improvements in successive quarters kept both activities to enable more interactions for individuals and student groups.
The engagement methods discussed in the first offering of the course were used to encourage students to participate and build a learning community in the online course from the very beginning of their learning, and this was continued in subsequent offerings of the course. The collaboration also focused on incorporating different pedagogical strategies to allow students to access the course materials and engage deeply with the course content (CAST, 2024) using different educational technologies.
While online teaching has the advantages of using multimedia such as videos and images to present course materials, it is always a challenge for individual instructors to create high-quality multimedia content due to the limited equipment and resources (A Pass Education, 2024). Closely working with the media production team, which is part of the instructional design team at UCLA, the instructional designer and the instructor well-planned and produced lecture videos in a professional studio environment and explored different ways that allowed students to interact and engage with those lecture videos. At each course offering, the ID/instructor collaboration brought new ways of representing the course content that supplemented the pre-recorded lectures. Although student satisfaction surveys revealed that students preferred to learn the course content with the studio pre-recorded lectures, they highly valued some of the external videos.
Learning Glass is a technology that allows instructors to write lecture notes on a transparent whiteboard paired with a filming system while recording. It maintains face-to-face contact with students and works well for writing formulas, demonstrating processes, and demonstrating math concepts. Mathematical learners often prefer it when their teachers solve problems by hand to show them the steps involved. Using Learning Glass to demonstrate how to solve practical problems and provide proof in the probability course is efficient and easy for students to understand the context.
In addition to maintaining the teacher’s presence through learning glass for online lecture videos, the collaboration also tried to recreate the nature of pause and reflective moments that usually happen in traditional classrooms and allowed students to reflect on what had been presented, the opportunity to ask questions or clarify on specific topics. Quiz-in-video (QIV), a feature of Kaltura, the video platform most departments use at UCLA, can easily recreate those reflection moments for online courses. Using QIV, this probability online course presented the key ideas and examples in segments along with knowledge-check questions that gave students an instant assessment of their comprehension and naturally prompted them to pause and reflect. From the survey responses, students considered the videos great educational value.
According to CoI, cognitive presence focuses on helping students connect and apply new ideas (CoI, 2021). There are different ways to motivate students to engage with course content cognitively, so in addition to pre-recorded lecture video segments for online learning, the course design selected and produced supplemental multimedia resources that use step-by-step problem-solving to demonstrate complex concepts. For instance, taking remarks from students regarding how hard a particular concept was or from the instructor’s prior knowledge, the instructor discussed the video length and what to cover with the instructional designer and then produced some just-in-time videos. Those shorter supplementary videos concentrated on a single exercise, supported particular lecture content, and helped students by explaining how to complete a particular probability task.
Besides the instructor-made just-in-time videos, this course also adopted materials from YouTube educational video content. In the era of digital learning, undergraduate students consciously participate in informal learning through social media exchanges and information absorption from others (Romero-Hall, 2021); in some research, students also could set up their own learner-initiated digital spaces next to their official classroom to facilitate their and their peers’ informal learning (CoI, 2021; Turcotte et al., 2021). The design of Introduction to Probability also included informal learning experiences for our online statistic learners. The course included relevant external videos to explain content as part of various learning resources.
In specific modules, activities asked students to create videos to help others understand specific topics. Students were encouraged to submit outside resources or produce videos independently to receive extra credits by sharing them in class discussion forums. The instructor and TA assisted students in authoring and editing their works to ensure accuracy and correctness.
To add to the depth of discussion, the course also asked guest speakers to share their insights and knowledge on certain complex concepts and ideas. The topics and content from guest speakers varied depending on students' and instructors’ feedback. Videos were included in the course from the beginning and were added in later offerings. However, students mentioned that the videos made by students were more beneficial than the ones made by guest speakers due to their direct connection with topics. In addition, students also shared in their feedback that while some found that using the supplemental materials made some concepts easier to understand after watching the formal lectures, others found that the lectures alone were also sufficient to understand the concepts.
UCLA requires upper-division courses to offer a weekly hour-long live discussion session for students within each quarter per course offering, usually led by a TA. Because the TA was different each quarter, creating structured and consistent discussion sessions for our course was challenging and a significant component of the ID/instructor collaboration for the course design. The design underwent several phases to maximize the time TAs spent with students during the discussion sessions. The course design for Fall 2021 aimed to involve TAs to engage students with additional readings, videos, and textbooks to help them reinforce their comprehension of the ideas covered in the pre-recorded lectures using supplemental materials that included probability applications and exercises that addressed any challenges in homework assignments and quizzes. However, some challenges remained due to the TA session scheduling.
Based on students’ feedback, the design for Winter 2022 adjusted the session schedule to enable students to review course materials before the session. Students were encouraged to bring questions regarding the readings assigned for the week’s module or the quizzes incorporated into the pre-recorded lectures and could review session recordings. However, attendance still did not increase, and some students wanted to discuss more complex questions.
The redesign for Spring 2022 incorporated a gamification approach. Students were asked to submit questions before the TA session via a Google Doc, and those attending and actively engaged with the session would receive tokens to exchange for other activities. A low-stakes, end-of-session graded quiz was used to gauge participation. The redesign successfully increased the quarter’s attendance rate to almost 90%, as well as students’ satisfaction.
For future redesign, we encouraged students to interact with the course material earlier in the week by adding one more formative quiz in the middle of the week.
According to UDL principles (CAST, 2024), the course design should allow multiple ways of action and expression, which encourages “using multiple tools for construction, composition, and creativity” (consideration 5.2) and “building fluencies with graduated support for practice and performance” (consideration 5.3). Within the developed information learning spaces, learner-initiated interactions and instructor-and-ID-generated activities always explore the opportunities for students to create and apply what they learn from the course materials. For instance, to support students fully comprehend the simulations and data visualization, some of which use what Gok & Goldstone (2022) called “Generic Visual Representations of Data,” including histograms and “Standard Simulation,” scaffolded projects were designed for our course to guide students to gather data and correlate it with the information presented in graphs through a variety of representations using complete examples in settings other than the project, additional videos explaining the process and instructor-produced videos showing steps. The following examples from the course design are described in more detail.
Students could stay in touch with the entire class throughout the quarter using the discussion forum in LMS. The fact that they were rated on a broad scale of A for significant contribution, B for some contribution, and C for minimal contribution partly explains the high level of participation. The open forum allowed students to assist one another with assignments, lectures, quiz questions, and occasionally even instructions. Over the quarter, fewer people participated in the open forum.
The UDL tenet of “Multiple means of action and expression” allows students to choose what goes into their assignments. For instance, during some of the weekly discussions in Canvas, students could opt to produce written pieces, films, photographs, recorded messages, or URLs of resources to demonstrate their understanding of the topics covered, support one another, or highlight specific areas of their major that are relevant to the course material. Then, to foster additional interpersonal interaction, each student would have to evaluate the contributions of at least three other students meaningfully.
Students found their peers’ contributions very illuminating. They were enlightened by their peers’ abilities to articulate concepts or opinions in a way that helped them better understand the course material, or they were shocked by how essential probability is in all of their majors, many of which were not statistics majors. Their comments made it possible for students to improve the themes of their conversations iteratively.
Scaffolding is an instructional method that gradually moves students to greater independence and understanding, allowing them to start at lower complexity and move to higher-complexity material. For a scaffolded project to succeed, learners must have the tools they need to do what is being asked of them, and tutorials on how to use the new tools must be provided. Our course used different scaffolded peer-reviewed projects where we opened the informal learning space, and the learners actively brought it to life for all to use simultaneously.
By allowing the peer-review approach, students were offered multiple opportunities to interact with each other and obtained a high level of ownership and accountability. In this course, using the students’ preferred media and Google Docs, each group created a small learning community digital cluster during the project’s production to collaborate and learn from one another informally. The students enjoyed these unstructured learning groups.
In one of the course’s scaffolded projects, ID/instructor created an unofficial space, like a Google Spreadsheet, where the groups, which were initially organized at the start of the quarter, collaboratively entered data gathered from a sample of 15 students per group from the same population. All information was collected through a survey-like questionnaire. Every group could view the data entered in the Google Sheet by other groups. There were two objectives for the data collection. First, to demonstrate how a sample was used to make inferences about a population in real-world contexts using all the group's data to represent a sizable sample from the population. Second, they used a separate spreadsheet where each group could enter the average for its fifteen sampled students and then the class used all these individual group averages to obtain a sampling distribution of averages. The class then, as a single group, determined whether the Central Limit Theorem held.
In comparison to the standard simulations of Gok and Goldstone (2022), students comprehended the concept of inference and the differences between the contexts of a sample’s histogram and the sampling distribution of means of multiple samples in a much more contextual way with the above experiment and multiple modes of media representation. The same procedure was used for other projects. However, students had to determine whether a given probability model adequately described the generated data, a “model fitting” exercise in the DS community. Weekly practice problems provided step-by-step instructions on demonstrating goodness of fit through examples.
While the collaboration between the instructor and the ID built the iterative process of producing a high-quality online course, learning theories and frameworks like CoI and UDL principles, student suggestions through feedback questionnaires assessed within the framework of the learning theories and the ID/instructor frequent evaluations and ongoing communication, all influenced the collaboration. The full-fledged partnership was beyond consultation and continued to enhance student engagement and collaborative learning quarter after quarter. The instructor perceived more social presence and collaboration among students in the FAOC compared to the in-person offering of the same course, owing to this collaboration. Students viewed the course as a unique and “high value” learning opportunity that enhanced their understanding of probability and other DS courses they had previously taken. Students' awareness that the class was a collaborative effort between the instructor and ID contributed to their positive opinion of the course. Furthermore, the ongoing communication with the ID and the iterative course redesign generated a continuous, highly customized, and transforming professional development experience for the instructor.
We still seek the opportunity to enhance the learning process regarding collaboration and engagement. Because some students believe that the supplemental external videos and materials are not part of the “formal” course, further offerings will change that perception by taking into consideration creating conditions to proactively encourage students to curate digital spaces for their learning, as suggested by Turcotte et al. (2021) and replacing some supplementary materials with students’ videos. Furthermore, TA sessions will be redesigned into a format more consistent with the remainder of the course; guest lectures will be more aligned with the student projects than with individual concepts in the course; more alternative ways of rewarding students for their participation will be offered, as we observe that there is overwhelming participation only when there are some points assigned to it.
Correspondence concerning this article should be addressed to Sirui Wang, UCLA Teaching and Learning Center, University of California Los Angeles (UCLA), 70 Powell Library Building, Box 951515, Los Angeles, CA 90095-1515, United States. Email: swang@teaching.ucla.edu
EdTech Archives