Instructional design teams, especially in online higher education, are challenged with achieving scalable course quality. According to the National Center for Education Statistics (2021), in fall 2021, 61% of undergraduate students and 56% of graduate students were enrolled in at least one distance education course, and amongst graduate students in private nonprofit universities, such as in this study, 29.8% were enrolled in fully online degree programs (NCES, 2020). With an increasing number of online programs and degrees offered, as well as meeting the required standards for accessibility, student engagement, and program alignment, the need for structured and sustainable quality assurance (QA) processes is clear. However, quality assurance frameworks have not evolved as quickly as the increased prevalence of online programs (Hinck et al., 2018). A typical approach to QA is to implement a comprehensive end-of-course review in which pedagogical, technical, editorial, and accessibility evaluations are merged into one feedback cycle. Although this model seems efficient, it can cause an overload in reviewers’ cognitive capacity at the end of development, resulting in unnecessarily lengthy and repetitive reviews and an inconsistent application of standards between multiple reviewers. Sweller et al. (2011) posit that “another category of cognitive load that requires working memory resources is imposed not by the intrinsic structure of the information but rather by the manner in which the information is presented” (p. 57). Through this lens, the previous model presented information that was unnecessarily taxing the cognitive load of the reviewer.
To resolve these concerns, our team redesigned our QA process, as shown in Figure 1, into a modular and multi-layered system to reduce the extraneous cognitive load presented to a reviewer (Sweller et al., 2011). This system allows for task distribution across multiple, specialized reviewers, while embedding QA review earlier and more consistently into the course building process. This article discusses the redesign from a practitioner’s point of view, describing the various strengths and challenges encountered and providing other instructional design teams with the outcomes for this system.
Figure 1
The EDGE QA Timeline
In this new process, we have divided the QA into multiple layers, which involve several specialized stakeholders at each level. These roles are detailed in Table 1.
Table 1
QA Stakeholders
Role | Role Description | Core Responsibilities in QA Process | Typical Review Time | Artifacts Produced |
LDM (Learning Design Manager) | Manages team of LxDs and oversees their work | Pedagogical review to ensure consistency within academic programs | 2 days | QA rubric |
SME (Subject Matter Expert) | Provides contextual content from a real-world practitioner perspective | Content outline review | 7 days | |
Peer LxD (Learning Experience Designer) | Designs and builds course content in LMS | Learning design content review using Quality Matters rubric | 2 days | QA rubric |
Media (Specialist media personnel, including Video Producers and Graphic Designers) | Creates all media for courses, including video, animation, and interactive graphic content | Review of video content, animations, interactive graphics | 1 day | |
LS (Learning Solutions) | Provides quality assurance for courses before launch and during course run | Technical and accessibility reviews | 5 days | - UDoit report in Canvas - QA rubric and report sent to LXD |
Our team is an online instructional design and implementation unit at a larger, private R1 institution. The team consists of learning experience designers, multimedia and studio team specialists, program/course implementation managers, and data/technical experts. The previous quality assurance (QA) system reflected a lengthy and less sustainable end-of-development model. One reviewer was responsible for all pedagogical, editorial, technical, and accessibility components in one round. While this model was created with good intentions, it overestimated what reviewers could accomplish within a fast-paced timeline and created gaps leading to inconsistent reviews. To get the work done, the reviewer may simply check boxes for completion without checking the content.
The redesign of QA came as resourcing became more difficult to assign to tasks, and the team was ultimately tired of the lengthy checks. The inefficiency of finding errors to correct at the end of a course development also led to a necessary redesign of the process. As Simunich et al. (2022) note, the success of quality assurance initiatives depends on the effective implementation of the processes, resources, and tools created to ensure quality, which was a gap our team sought to address.
The reviewers were each responsible for a single exhaustive checklist that included the following criteria:
Clarity and grammar
Coherence of the assessment and alignment with learning materials
Navigation of the LMS
Functionality of all links and media
Compliance with accessibility standards
Table 2
Challenges With the Previous Model
Challenge | Description |
Cognitive and role overload | Reviewers found it challenging to stay accurate across domains, particularly when switching back and forth from pedagogical to technical issues. Additionally, accessibility tests had the risk of being missed due to last-minute QA, resulting in not fully testing courses on all necessary platforms (mobile, tablet, browsers, etc.) with different tools (screen readers, color contrast checkers, etc.). |
Extensive feedback | Reviewers tended to provide long lists of revisions to address right before concluding the course, leading to frustration and rushing through the ability to make changes. |
Inconsistencies in standards | Reviewers took different approaches based on their comfort level with accessibility and technical features. Feedback was of varying quality. |
An analysis of workflow problems | Because of the late addition of QA review and the need for highly specialized QA review, one late review could delay the whole course timeline. |
These problems indicated the requirement of a more distributed and manageable approach.
The revised model splits QA into four layers, with each layer assigned to a reviewer or team member with a specific set of skills.
Copyediting: clarity, tone, grammar, consistency
Pedagogical QA: alignment, engagement, clarity of objectives, module coherence
Technical QA: navigation, LMS structure, link checking
Accessibility QA: alt text, captions, color contrast, document structure
Rather than reviewing the whole course at once, each module is submitted for QA review separately. This arrangement enables quick fixes, minimizes the length of revision cycles, and evenly distributes the workload across the course development timeline.
Each module is tracked across the various stages of the QA process. This is recorded on a shared tracking sheet so that designers and reviewers can see the module’s status as it is submitted, in progress, and completed, leading to more efficient communication between teams.
The benefits of the new model as outlined in Table 3 are based on qualitative feedback we received from the team after implementation. This was informally gathered in meetings with those involved in completing QAs.
Table 3
Benefits of the Redesigned QA Model
Benefits | Description |
Improved clarity and focus | Reviewers focusing on one domain at a time appreciated the increased focus as well as the reduced cognitive load and the confidence that resulted from their evaluations. |
Faster and more actionable feedback | More manageable, incremental feedback was provided, instead of longer, consolidated feedback given at the end of the development cycle. While hard data cannot be provided to show this improved time due to lack of data from the original process, qualitative data after speaking with impacted team members proved a more manageable and timely process. |
Better workload distribution | The modular approach created more predictable resource planning and removed last-minute bottlenecking. |
Greater team specialization | The opportunity for reviewers to focus on and gain increased domain-specific expertise led to improved review quality. |
This modular approach helps with cognitive capacity, workloads, and timely feedback, but it also needs to be handled in a structured way to ensure enough resources can be allocated. Additionally, sound project management practices and a clear tracker are essential for transparency in when reviews will be completed and by whom.
The redesign introduced several new complexities that demanded continuous iteration. This information was similarly gathered, in an informal way, with team members who were active participants in the new model.
Table 4
Challenges of the Redesigned QA Model
Challenge | Description |
Effort in coordination | A multi-layered system calls for seamless coordination and synchronized visibility. If maintenance is not done regularly, the system can become stagnant. |
Reviewers’ content knowledge | With a set of reviewers, there can be varying levels of understanding when it comes to the content being reviewed. |
Overall lengthening of the timeline without project management | While there is a greater capacity for feedback, there is potential to extend the overall timeline required for a project as the number of review cycles is cumulative. To avoid this and complete tasks quicker than the previous model, it is important to have a project management effort in place. |
Change management | The staggered feedback was an initial adjustment required by all involved, as developing designers had been used to receiving all course feedback at once. |
As the implementers of this new model, we have developed a few practitioner-oriented insights as a result of this redesigned implementation.
Table 5
Lessons Learned from the Redesigned QA Model
Lesson | Description |
The redesign of the workflow is best paired with improved project management and resourcing | With transparent timelines, tracking tools, and shared expectations, module review can be done efficiently without delays. |
Specialization can improve quality and speed, but cross-training ensures resilience | We know that teams can get more done when everyone is able to develop individual skills to their fullest, but cross-training is actually more effective when it comes to preventing disruptions to workflow. It allows for a more resilient workflow overall if there are team member absences or other unforeseen circumstances by having others cross-trained as a backup resource. |
Process review meetings are essential | Opportunities to give and receive feedback, talk about cases, and realign on criteria help reviewers review work more effectively. |
Early communication reduces confusion | Setting clear expectations about how and when you will provide feedback helps reduce frustration and increase buy-in. |
Start small and scale | Starting with one course, and then with one program, allows the team to refine their process before scaling it to a broader application. |
Qualitative insights are meaningful | While it may seem that quantitative data are the only valid measure, feedback on themes from the designers and QA reviewers are all indicators that valid insights are present. |
Looking at the process of Quality Assurance through a more layered and modular lens allowed us to separate the copyediting, pedagogical, technical, and accessibility reviews, giving us a more sustainable and collaborative workflow. Reviewer overload was reduced, and the feedback was easier to digest, but it did create a need for more coordinated effort between teams. To help coordinate this effort, project management tools like Smartsheet were implemented to keep track of all requests and mark completions. The work showcased communication, role definition, and process refinement.
As institutions scale their online learning offerings, QA processes demand flexibility, effectiveness, and a careful understanding of the realities of instructional design practices. The lessons learned in this implementation case through this workflow redesign can support other instructional design teams seeking to balance rigor with practicality in their approach to quality assurance.
EdTech Archives