Instructional Design (ID) as a profession has grown in recent years, with an expected growth of 10% from 2020 to 2030 (Prusko & Kilgore, 2023). Due to the COVID-19 pandemic and the subsequent shift to virtual instruction in schools and work environments (Petherbridge et al., 2022), many institutions within educational and corporate landscapes have accepted and integrated the idea of online learning and training into common practice. Because of the knowledge needed to create instructional materials, there is an increasing demand for learning and development practitioners (Posner, 2023). ID training programs need, more than ever, to support novices as they transition into the field and bridge the gap between theory and practice.
With a focus on the development of novices’ skills in e-learning development, suggestions for the improvement of instructional experiences can arise from the analysis of design judgments (Lachheb & Boling, 2021; Nelson & Stolterman, 2012) made during the development of an e-learning project in an instructional setting. This study provides a second-order narrative (Creswell & Poth, 2023) account of the experiences of six novice e-learning developers during the creation of a branching e-learning module. Participants created their modules in the context of an online asynchronous graduate-level class focused on learning e-learning software and evidence-based principles for effective e-learning design set forth by Mayer (2017) and Clark and Mayer (2016), as well as relevant graphic design principles (e.g., Williams, 2015). These experiences, including beliefs, obstacles, and solutions to problems, result in important recommendations for instructional spaces and approaches to support novice Instructional Designers (henceforth IDs) and e-learning developers.
This article is structured as follows: First, we present a literature review discussing design judgments, what they are, and why it is important to study them, in addition to reflection during and after design and development. We then describe the methods, including our context and analytical procedures. Following this section, we present the results of our study, including the design judgments made by participants in each phase of e-learning project creation. Next, we discuss these results, contextualizing them in relation to previous studies and suggesting instructional strategies based on the chronology of design judgments identified. Finally, we conclude the article with a summary, limitations, and recommendations for further investigation.
While the concept of design judgments is not new, their intersection with instructional practices demands further attention. Design judgments are unconscious ideas that are thought to be drawn from past experiences (Farmer & Koehler, 2022). These judgments are a part of the essence of what makes a designer a designer and are based on intuitive and rational logic (Korkmaz & Boling, 2014; Nelson & Stolterman, 2012). Specific to the context of this study, in which novice IDs create a specific instructional module, research has identified that these professionals use design judgments in making design decisions (Farmer & Koehler, 2022).
When understanding design judgments, it is equally important to understand what they are not. Nelson and Stolterman (2012) argue that judgment is not necessarily a rule of logic, but the “accumulation of the experience of consequences from choices made in complex situations” (p. 181). Though design judgments can sometimes be mistaken for design decisions, judgments deal more with the why than the how. As such, design judgments are integral to the decision-making process when creating instructional materials.
Eleven design judgments can be invoked during specific situations, and judgments do not necessarily occur in isolation (Lachheb & Boling, 2021; Stefaniak, 2023). These design judgments are: core, which refers to a designer’s beliefs and values that can influence or drive other judgments; instrumental, the selection of practices and tools for the intended design; framing, the establishment of the specific components and limits of the design; default, intuitive responses to design situations; deliberated offhand (DOH), intentional application of default judgment; appreciative, the special focus on specific design elements; quality, an assessment of the artifact based on known principles and practices; appearance, which relates to an evaluation of the design as a whole; navigational, which drives decisions regarding process according to the desired goals; connective, the correlation and assessment of alignment of design or objects; and, finally, compositional, which drives decision-making related to the assembly of design components in an ideal manner. Judgments are complex and can evolve, sometimes through discussion or brainstorming activities (Demiral-Uzan, 2015). Design judgments are clustered together because of their connections (Boling et al., 2017) and are context-specific with the goal of reducing complex situations or problem-solving.
Historically, IDs (those responsible for entire instructional design processes, from analysis to evaluation) and e-learning developers (those specializing in design and development) are largely taught to use sequential formalized models and theories. The ADDIE model, for instance, is a systematic approach that uses five phases (analysis, design, development, implementation, and evaluation). Given its well-defined nature, in theory, one phase starts upon completion of the prior phase, and at the end of the process, a finished product is created (Aydin et al., 2023; Muruganantham, 2015). If interpreted linearly, the ADDIE model, similar to other ID models, may not capture the complexity and breadth of situations faced by designers in practice (Smith & Boling, 2009).
Interpreting models as linear approaches can be ineffective when an ID needs to make quick decisions when dealing with specific constraints, assisting clients, or working with other stakeholders or project team members (Lachheb & Boling, 2021). In real development contexts, the ADDIE model should be implemented iteratively (Peterson, 2003) and nonlinearly according to the demands of the instructional context (Larson & Lockee, 2019).
In such complex, nonlinear design processes, design judgments are unconsciously utilized. IDs who find success in using such judgments as their primary rationale tend to follow irregular decision-making processes that most commonly mimic what IDs experience in their day-to-day jobs (Farmer & Koehler, 2022; Gray et al., 2015; Demiral-Uzan, 2015; Korkmaz & Boling, 2014). As an ID becomes more experienced, they might consciously enact design judgments via “rules of thumb” (Ertmer et al., 2009, p. 19) as they adapt models and processes according to project guidelines and established constraints.
Because traditionally trained IDs are primarily taught theory, models, and design principles rather than design judgments, they may not know how to hone in on certain design judgments that could benefit them in circumstances of ambiguity (Lachheb & Boling, 2021). A potential way this can be combated is to make design judgments known during educational experiences. If IDs are taught to be aware of their own design judgments, they can use reflective practices to further understand whether certain judgments have benefited them. If judgments are proven to result in effective design decisions, such judgments can then be understood, developed, and consciously utilized (Gray et al., 2015), potentially leading to the establishment of rules of thumb (Ertmer et al., 2009). The analysis presented in this paper maps the judgments made by participants in a continuum, thus contributing to a growing understanding of how design judgments are enacted in complex, nonlinear design processes.
Identifying one’s design judgments can be facilitated by reflection both during and after the creation of a project. That said, the following section discusses reflection and its benefits to the design processes and the purposes of this study.
Schön (1983) has identified two primary types of design reflection: reflection-in-action and reflection-on-action. The primary differences between these reflections are their timing, nature, and purpose. Reflection-in-action involves the inner, unconscious discourse that designers experience while actively problem-solving. In contrast, reflection-on-action is a more retrospective analysis, conducted after the completion of a design project, typically within the post-mortem phase, where lessons and insights are extracted (Lachheb & Boling, 2021; Schön, 1983).
Both design reflections are valuable strategies for improving professional competencies and expertise, including the ability to address complex problems and facilitate learning from personal experiences. When combined with supportive mentors and supervisors, reflection can enhance the learning process, improving understanding of complexity and transfer of theoretical knowledge to practice (Mann et al., 2007). When explicitly addressing reflection in instruction, however, it is important to establish guidelines and emphasize the importance of this activity for professional development (Fessl et al., 2017). Considering these recommended practices, participants in this study were encouraged to problem-solve specific design and development tasks themselves and externalize their thinking in reflective discussion activities at the beginning, middle, and end of project development. Analyzing data from these participants’ reflections can shed light on critical moments during project creation in which students reflect in and on action.
This qualitative case study (Moore et al., 2023) aimed at identifying and mapping the design judgments of six novice e-learning developers within the context of a Master’s level class on e-learning development. Participants’ design judgments were to be identified from the idea creation stage to the development of e-learning modules and supporting design documents to inform instructional strategies for teaching e-learning design and development and supporting learners throughout the creation of e-learning projects. To achieve this broader goal, we address the following research questions:
Data was collected in two offerings of a 10-week online asynchronous Master’s level course focusing on design principles and e-learning technical skills. This course adopts a resource-based learning approach (Hill & Hannafin, 2001), where students address authentic and simulated workplace performance issues using specialized e-learning software. This class is offered in the Canvas Learning Management System within the context of a Master’s and Certificate program for current and aspiring IDs, e-learning developers, and other performance improvement practitioners in a university in the Northwestern United States. In these ten weeks, students develop an instructional video (Weeks 2-3), a web-based online module using Articulate Rise (Weeks 4-5), and an interactive module that includes a scenario-based learning component (Weeks 6-9). This investigation focuses on the projects created by students in weeks 6 through 9. These projects were created using the e-learning development tool Articulate Storyline. Students were allowed to purchase a subscription to the software or use Storyline’s available 30-day free trial for project creation. Out of 30 invited students, six volunteered to participate in all of the data collection steps of the study. Volunteer participants’ reported demographic data and information regarding experience with tools and e-learning development are presented in the table below.
Table 1
Participants informed demographic data
Pseudonym | Age | Gender | Ethnicity | Experience |
Brian | 45- 50 | Male | Caucasian | Prior experience with e-learning planning and design and tools associated with these processes; Prior experience with and knowledge of basic functionality of Articulate Storyline |
Robert | 40-45 | Male | Latino/Hispanic | No prior experience with e-learning design and development |
Amanda | 35-40 | Female | Caucasian | Prior experience with and knowledge of the basic functionality of Articulate Storyline |
Beth | 40-45 | Female | Caucasian | No prior experience with e-learning design and development |
Laura | 40-45 | Female | Caucasian | Prior experiences with e-learning planning and design and tools associated with these processes; Previous experience with e-learning tools other than Articulate Storyline and their functionality |
David | 30-35 | Male | Caucasian | Previous experience with e-learning tools other than Articulate Storyline and their functionality |
The data obtained from each participant for this study was as follows: an initial draft plan posted in the discussion board in Week 6 of the course; a project progress report posted in the discussion board in Week 8 of the course; a design document and reflection submitted upon completion of the project at the end of week 9 of the course; and a semi-structured interview conducted after the completion of the course. Data was collected throughout the participants’ development of the course's final project, which consisted of a 10–15-minute e-learning module that addressed a performance need identified by students. Due to the time constraints of this asynchronous course, participants were also the subject-matter experts for the modules created. Though students selected their topics for their personal projects, the assignment guidelines specified that the created module must contain a linear instructional component and a branching scenario component. The guidelines also required students to apply design principles such as C.R.A.P effectively (Williams, 2015) and Mayer’s (2017) principles for using multimedia in e-learning as well as accessibility features (i.e., World Wide Web Consortium, 2023).
The first step of data analysis followed a deductive approach (Elo & Kyngäs, 2008). Data was first coded based on previous definitions and research on design judgments (Lachheb & Boling, 2021; Nelson & Stolterman, 2012). The predefined codes were based on the eleven types of design judgment, in addition to reflection in and on action. The code definitions for each design judgment used to guide the analysis mirror the definitions presented in the literature review section. When analyzing data, two researchers first coded data separately. Then, codes were compared and modified, if needed, after discussions until the researchers reached an agreement. These codes were chronologically plotted (Nasheeda et al., 2019) based on different steps of project creation established by the researchers. These steps were based on the progression of project development established in the ADDIE model with modifications due to the context and purpose of the study. The steps are as follows: Idea creation and goal setting; Design; Learning and exploring authoring tools; Development; and Experience and Learning Object Assessment. Other codes pertained not to one step but to the entire design and development process, thus being categorized as such.
In order to represent the process of making design judgments throughout participants’ project creation experiences, we present such judgments in a second-order narrative (Creswell & Poth, 2023) that addresses our research questions. This type of narrative allows researchers to organize people’s individual or collective experiences as a story. For this study, we have developed a collective narrative based on the commonalities of participants’ experiences during their e-learning project development.
This study has been approved by the researchers’ university Institutional Review Board. To ensure confidentiality, all identifiable information pertaining to participants and instructors has been removed from the data, except for demographic and experience information used to contextualize students' backgrounds. Pseudonyms identify all participants throughout reporting. Before data collection, participants signed a consent form informing them of their rights throughout the research process, including the possibility of not responding to any questions that could generate discomfort during the semi-structured interview. Finally, this article was sent to participants after it was finalized to address potential confidentiality concerns before publication. Participants also provided valuable feedback based on their developed design expertise.
In this section, we present the results of our study in a narrative format. We discuss the identified design judgments in each project creation phase and their nature to capture the complexity of participants’ collective project creation experiences. The narrative provides examples of codes to illustrate the overarching themes identified within each design judgment when applicable. The figure below shows the number of design judgment codes identified for each participant across all data, and the narrative follows.
Figure 1
Number of coded design judgments by participant
As Week 6 of the course starts, participants define their ideas for a learning object (Chyung, 2007), including a declarative/procedural component and a branching scenario. Over the previous weeks, students had been introduced to design principles and practices that would support them in this task. Unlike previous projects in the course, this task was less defined, requiring students to choose their subject matter, establish learning objectives, and define scope. The level of reflection required in this brainstorming phase led to a myriad of framing judgments (53 instances), the establishment of the specific components and limits of the design, in the phase categorized as Idea creation and goal setting. Study participants set the topic and constraints for their projects based, for instance, on their workplaces. David states that his e-learning module will target “IT technical support staff who are new to helping support virtual exams that medical students take at the end of each required clinical rotation.” Robert’s project, in turn, addresses the needs of “many employees over a large geographical area and [can be] taught to multiple levels of employees.”
Participants also frame their projects according to a proposed structure. When discussing the details of her proposed learning experience, Amanda says that her module
has a directive segment that exposes learners to the most basic information, examines a commonly misunderstood section in a more practical sense (opening contacts), and then uses scenario-based eLearning to promote critical thinking skills with more context for use on the job.
Framing judgments can also be found in participants’ conceptualization of the nature of their scenarios, including establishing content complexity. In her proposed project, Laura describes learners make “choices whereby either solution may not be the best option (ethics over politics) yet is the most realistic in the workplace. Simply, the learner might have to select the lesser of two evils.”
Such complexity was a deciding factor for pursuing one project idea over another when multiple options had been generated. Considering the time constraints for realizing a project idea, Beth opted to decrease the complexity of her module by prioritizing an idea that led to well-defined decision-making:
I originally was going to do emotional intelligence. But when I started trying to create the branching scenarios, there wasn’t really a clear right or wrong, so it was really tricky to write the feedback. So, I opted for something simpler, with “feedback” as my topic.
When choosing projects with a high level of complexity, framing was needed to adjust the scope to a defined idea that could result in an e-learning project that would be completed in less than a month. In his interview, David described his process for deciding on the specific goals for his project within his overarching topic:
It took a little bit of time for me to kind of center on that and keep that focus. Because I kind of want to go and, oh, what about this type of scenario? There were differences between, if students were using Macs versus PCs, and of how I approach things, too. And so, realizing that, again, okay, I just got to stick to one topic, one type of device for a scenario especially, will be useful.
During the idea creation process, participants also made many core judgments (52 instances), referring to their guiding knowledge and beliefs about e-learning design and development as well as the characteristics of their target audience. Based on her expertise in scenario-based e-learning, Amanda proposes her design include “a scenario portion that will accelerate assistant expertise without exposing x-rays on real patients.” Laura also draws upon her previous knowledge and beliefs regarding the effective combination of principles and practice: “Through the use of Articulate Storyline... the content developer is able to create a well-developed learning experience that perfectly marries technology, pedagogy, and content knowledge.”
Instrumental judgments (22 instances), the selection of practices and tools for the intended design, are also notable in the Idea creation and goal-setting phase. Participants could choose between slide-based e-learning authoring tools within the project guidelines. However, all six participants considered learning to use Articulate Storyline as the better choice based on evidence from job postings and previous interactions with practicing IDs and e-learning developers. Having chosen Storyline as their main tool, participants then made decisions regarding supporting tools based on the characteristics of their project. Amanda justified her planned use of videos originating from the social platform TikTok considering potential learner profiles: “TikTok videos were used to gain attention of the target audience as the group is very active in the social media platform.” To personalize the look and feel of his module, Brian ventured to explore the Artificial Intelligence image generation tool Midjourney to create characters for his scenario. Instrumental judgment can also be found in using tools to support design. Laura, for example, discusses the traditional tools used for planning and designing her project: “paper and pencil, PowerPoint as my image board with the ability to move the slides up and down with ease, and Word to gather my thoughts on overall concepts."
Having made framing and instrumental judgments, which were directly influenced by core judgments (designer’s beliefs and values that can influence or drive other judgments), participants made navigational judgments (20 instances). These judgments drive decisions regarding the process according to the desired goals. In this phase, participants enacted navigational judgments when deciding how to learn how to use the authoring tool and establish design and development processes. Demonstrating her dedication to thorough planning and design, Beth discussed different design steps in creating and refining her idea and project design:
I scripted out all the actual content I wanted covered in the training. I crafted learning objectives. I built it out in PowerPoint/Canva to help me visualize the flow of information and where there are opportunities of engagement. I created a storyboard of my entire project so that I could see how it would flow.
Other participants also described their navigational judgments to ensure the success of their projects later on. Laura chose to explore Articulate Storyline thoroughly before designing her module to learn about the possibilities and constraints of its features. David also adopted this approach, referring to resources available on the web to aid his learning process in using the specific functionality of the e-learning tool based on the needs of his project. The adoption of this strategy reflects default judgments (9 instances), intuitive responses to design situations, made by these two participants who had previous experience with e-Learning development tools.
Participants proceeded into the design phase after solidifying their concepts and establishing project goals. A central theme during this stage was devising strategies to plan and structure learner content systematically. Participants exhibited a clear sense of meticulousness throughout this stage while acknowledging the need to adapt when the process did not unfold as originally planned. This theme of deliberate foresight manifested in numerous decisions made during the design phase, with navigational judgment being the most frequently employed (28 instances). A common decision related to this type of judgment was using planning documents to guide the design and development. For instance, Amanda utilized storyboarding as a form of early planning to maintain focus:
Planning the module using storyboards [in another class] has proven helpful because I can break the project into a specific number of slides and understand the end goal before using Storyline to bring it to life. Even though I do not have enough slides to share for this update post, I feel using the storyboard slides has helped me remain focused on the end goal.
Other participants who did not use traditional storyboards still created a general plan for their scenarios, employing, for example, design documents or flowcharts. The primary rationale behind this intentional planning was to decrease their confusion while planning learners' different pathways. Many participants mentioned the need to emulate real-life scenarios, resulting in several different outcomes learners could take (the “best,” “okay,” and “poor”). Navigational judgment was also evident in other content-related decisions. Some participants emphasized using visuals to convey concepts, while others relied on on-screen agents.
By this phase, participants were already experimenting with a myriad of tools. Due to the technical nature of this assignment, instrumental judgments were also common (20 instances). Participants continued using word editors, slide presentation software, and flowcharting tools. Some participants, such as Beth, used a “mixed media” approach, where multiple types of planning artifacts and tools were utilized, depending on needs: “I use Canva. And I created the slides that flowed from my content, and that helped me generate ideas for the engagement piece and the interactions. So, then I used that to create my design document.” Similarly, David used various tools, starting with Google Docs, and eventually planned out the intricacies of learner pathways using LucidChart. David spoke about how LucidChart helped his mapping process when he found himself needing to adjust the scope of his project:
I did use LucidChart for the scenario to start outlining the scenario, what it could look like… here are the two flows and things like that. And that's where doing that and seeing how quickly it scattered in the tree branch was something very quickly I realized… I got to pull this back.
Default judgments (13 instances) were also present when participants chose tools to work with. Brian, David, and Laura started their planning process using tools they had prior experience with, either through design projects or other school tasks. It was not until participants needed to map out learner pathways that they began using tools they were less familiar with, primarily the flowchart tools mentioned earlier. Having more experience than most participants, Laura had the most prevalent number of default judgments (5 out of 13) and was the only one who mentioned intentional planning around evidence-based design processes and methodologies. Others, such as Amanda, also used a past class experience to determine her workflow process, mentioning she had previously learned the importance of having a “solid plan on design elements” prior to development.
While planning has been intentional and conscious until now, instances of framing judgments (12 instances) spoke towards the need to adapt on the spot, specifically regarding decreasing project scope, an aspect where most participants struggled. Ultimately, participants found that creating simpler projects was the correct way to proceed. Amanda states that she
started out with a big, elaborate plan, and quickly found out that I wasn't going to be able to tackle all of those with this one project. So, I just focused on one simple one that I think would help the most in the [place] I work at.
In the design phase, core judgments (11 instances) focused on the learner’s experience. Beth wanted to decrease the potential for extraneous cognitive load. Brian and Robert described using unique images to create a personalized learning experience. David and Laura both tapped into previous experiences designing scenario-based learning to strive to create meaningful learning journeys rather than modules with “superficial” designs. Deliberate offhand judgments (11 instances) also speak towards this same theme. Participants could utilize past successful experiences through past classes or designs to create higher-quality learning experiences by selecting both design approaches and tools.
While in the design and development phases, participants also advanced their knowledge of the tool chosen for e-learning development, Articulate Storyline. Though judgments that pertain to upskilling are not traditionally considered in design judgment research, the process of learning how to use Storyline was, in the experience of these novice participants, an integral part of design and development. Even when participants were already familiar with the tool, such as in the case of Brian and Amanda, further learning was necessary to meet the requirements of the activity, particularly as it pertained to interactivity and accessibility features. This phase consisted primarily of navigational judgments, with 23 instances. Participants referred to video guides provided by the instructor in addition to other tutorials found on the web for guidance on performing actions within the software. Oftentimes, participants followed the tutorial step-by-step, as exemplified by Beth’s statement: “the way I learn, I can't watch the whole tutorial, and even remember what they were trying to do. So, I would pause it and then try to figure it out.” Online resources were valuable as participants attempted to incorporate specific interactive features into their designs beyond the functionality covered in instructor-provided videos. Robert states: “I would look online, look for those videos. And then I would watch the video and then go back into Storyline and try to accomplish the task.” In other cases, tool experimentation was also used to establish directions and constraints for functionality and overall design of the planned module. For instance, David described using multiple resources simultaneously to assist him in flowcharting his module.
In the learning process, participants also made deliberated offhand (7 instances) judgments, intentionally applying default judgments, as well as instrumental judgments (7 instances) based on previously successful software learning experiences. Participants referred to web searches and videos freely available on video-hosting websites as resources that assisted them in implementing required or desired functionality into their modules. These resources provided valuable and highly specific information:
…usually, videos help me the most because they are very straightforward… most Youtube videos are synced up with that exact topic that I'm looking for. So, I don't have to scan through a whole 30-minute video. It usually would give me like 2 or 3 minutes of instructional purposes that I needed. (Brian)
Participants employed different strategies while learning the tool that facilitated their development phase.
Students started developing their projects within the e-learning development tool during and after the design phase. Having decided upon software and resources to aid them in accomplishing their design goals, participants created their modules based on previous planning, adjusting scope and structure based on their experience learning the software and time constraints. In this development phase, similarly to the design phase, navigational judgments were the most recurrent (42 instances).
Navigational judgments pertained to, for example, deciding on practices to facilitate development. Amanda describes how using visualization features of the e-learning tool was crucial in her practice: “I did find myself using the Story, that main slide [view], a lot more, moving my slide tiles around to make sure that they're going in the right spots." Decisions regarding features, both developer and user-facing, were also influenced by navigational judgments. In her Progress Discussion post, Beth reflects on her use of standardized naming conventions for project slides: “I can see how much value that adds to a project by keeping with naming conventions… it would be helpful for another ID coming onto a project that I created and having to make changes." Furthermore, problem-solving was another common theme associated with navigational judgments. These judgments were often made following important quality and appearance judgments, which were also significant in this phase. Brian, for example, discussed an adjustment made so module progression would be simplified after he experienced difficulties setting up variables within Articulate Storyline. Upon realizing that his template did not fit his design vision, David manually made color and structural changes to his entire module. In her case, Laura added a black-and-white text box over most pictures in her project to decrease the possibility of contrast issues.
As previously mentioned, quality judgments were also common during development (30 instances), as participants recalled their assessment of the design according to established principles and guidelines. These judgments pertained to, for instance, ensuring enough contrast had been applied throughout the module. Robert stated: “ I carefully chose colors based on research. Specifically, I selected light blue and black text and dark blue with white text." Beth reflected on how her preference resulted in contrast issues: “I love yellow, so I wanted to use this as the theme color for my design, although that got me in contrast trouble later on." In keeping up with the theme of decisions that conflicted with guidelines, Brian described how his font choice caused alignment problems in his module that could not be resolved with software features. Furthermore, instrumental judgments also assisted in quality judgments. David reflects on his use of a tool to ensure visually impaired learners could navigate his module:
NVDA was used to help ensure a screen reader could interact with the course appropriately. I think I have it set, but I saw some times where… Even where I thought I [had] designed for screen readers, I saw moments in the scenario where it can especially be difficult to ensure appropriate organization.
Other instrumental judgments (19 instances) made during development aided the decision-making for tools helping participants create or edit visual and auditory assets for their projects. David and Brian primarily reported instrumental judgments in this phase, as they experimented with and used video editing and other Artificial Intelligence (AI)- powered tools to generate and manipulate multimedia included in their modules.
Throughout development, but especially as participants finalized their modules, appearance judgments (16 instances), which relate to an evaluation of the design as a whole, were at the forefront. Participants assessed navigation and slide quality within the module to verify if progression and interactions were logical. Participants discussed appearance judgments within the context of software features, capabilities, and issues. Brian, for example, discusses how manipulating one object within a slide led to issues in other components. David experienced navigation problems due to branching with a combination of layers and slides. Moreover, Laura reflected upon functionality concerns from using multiple triggers in a slide. These and other issues required navigational judgments to problem-solve.
Finally, appreciative judgments (15 instances), the special focus on specific design elements, were also noticeable in the development phase as participants focused on visual, interactive, and accessibility components when reporting their judgments and decisions. Taking the instructional context into account, this focus on multimedia aspects arose from the focus of the class on technical skill development, in addition to the identified importance of the software learning process before and during project creation to ensure success.
After submitting their assignments, participants could reflect on their design process and overall experience. Experience and assessment data were primarily identified in interviews and design reflections. Reflection-on-action dominates this final stage (118 instances). Regardless of their experience level, almost every participant reflected on shared struggles around time management, particularly initial planning. They also reflected on time spent learning the intricacies of the tools needed to use during the development stage, particularly Storyline.
Speaking about the planning stage, Amanda mentions, looking back, she should have dedicated more time to development, but could not do so due to difficulties in planning: “For me the hardest part was just the planning. I spent too much time and effort on that, and I could have spent more time… perfecting my project before I turned it in towards the end.” Other participants expressed similar sentiments. Brian, for example, admits to placing too much emphasis on planning out module intricacies, such as character reactions, recognizing he was placing too much emphasis on minutiae. Furthermore, even with the common notion of believing they had taken too long to plan out their projects, Beth and Brian attributed their later success in mapping out their module in Articulate Storyline with relative ease, solely because of the flowcharts they had previously created.
When reflecting on their experiences in the development stage, participants did not initially expect to experience such high learning curves around the tools they would be using, particularly the e-learning software and accessibility checkers. Spending extended time on their planning and development stages meant participants needed to compromise. As noted in earlier sections, adjustments were made to the project scope, the complexity of learner interactions, and the creative output of objects to ensure project completion within the required timeline.
Some core judgments were also evoked during this phase (6 instances), each a nod to the unique experiences and key takeaways between participants. Even though Laura and Robert have different experience levels, they both noted how the experience of creating this e-learning project will positively influence future e-learning development. David also speaks towards “real world” practicality, emphasizing the difficulty in applying theoretical principles to design and development: “The differences between theory and practice, and how to apply theory in practice, cannot be underestimated.” Brian, however, had a completely distinct perspective. After using generative AI tools to create his media, he mentions the importance of creating “unique identifiers” to provide better experiences for learners.
Because participants were given opportunities across different data types to reflect on their experiences and final projects, many provided examples of what they would change if they were to revisit their projects in the future, demonstrating iterative thinking. Comments regarding potential future changes were made through instrumental (4 instances) and navigational (4 instances) judgments. For example, Beth stated she could more effectively use Twine to design scenarios. At the same time, David discusses that he would be able to apply better practices in recording narration.
Our results indicate that novice developers' e-learning development processes can be permeated by various design judgments manifesting at separate times during development. What we categorized as the “idea creation and goal-setting phase” involved substantial framing judgments, with participants defining their subject matter and objectives. This process required high levels of reflection and creativity. The “design” phase highlighted meticulous planning and the need for adaptability. In this phase, navigational judgment was predominant, with participants focusing on systematic planning and structuring learner content. These judgments were also prominent in the “development” phase, with participants focusing on problem-solving and quality assurance in their e-learning module designs. Throughout this process, students developed their knowledge of Articulate Storyline, integrating navigational, deliberated offhand, and instrumental judgments to enhance their technical skills. Table 2 illustrates the notable design judgments enacted while participants learned to use Storyline during project creation.
Table 2
Notable design judgments enacted in the “Learning the tool” phase
| Judgment | Codes |
|---|---|
| Navigational | 23 |
| DOH | 7 |
| Instrumental | 7 |
| Default | 4 |
At the end of the process, participants assessed their experience and their learning objects. This final phase was dominated by reflection-on-action, focusing on the challenges of time and project scope management and tool mastery. Table 3 illustrates the most common judgments made by participants in each step of project creation, along with the number of coded instances for each reported judgment.
Table 3
Chronological mapping of participants’ notable design judgments (from left to right)
| 1) Idea creation and goal-setting | 2) Design | 3) Development | 4) Experience and Learning Object Assessment | ||||
|---|---|---|---|---|---|---|---|
| Judgment | Codes | Judgment | Codes | Judgment | Codes | Judgment | Codes |
| Framing | 53 | Navigational | 28 | Navigational | 42 | Reflection-on-action | 118 |
| Core | 52 | Instrumental | 20 | Quality | 30 | Core | 6 |
| Instrumental | 22 | Default | 13 | Instrumental | 19 | Instrumental | 4 |
| Navigational | 20 | Framing | 12 | Appearance | 16 | Navigational | 4 |
| Default | 9 | Core | 11 | Appreciative | 15 | DOH | 3 |
Design judgments identified can manifest outside the constraints established by ID models and processes. For example, though framing judgments are prevalent during idea creation and goal setting, they are also present during design. Since participants’ projects were individual endeavors, the distinction between the design and development phases was also often blurry, with the added variable of learning and exploring a new authoring tool during these phases. Table 4 illustrates design judgments that were enacted throughout design and development.
Table 4
Notable design judgments enacted throughout the “Design” and “Development” phases
| Judgment | Codes |
|---|---|
| Navigational | 17 |
| Framing | 7 |
| Instrumental | 7 |
| Connective | 6 |
| Quality | 5 |
Our findings align with the unpredictable nature of e-learning design and development processes previously reported by literature (Farmer & Koehler, 2022) and the frequent changes in design judgments made throughout project creation (Demiral-Uzan, 2015). As is the case in design projects, participants were required to deal with uncertainty (Stefaniak et al., 2022), particularly as it pertained to design processes and scope, which was amplified by the 4-week time constraint of the activity. Additionally, the need to learn Storyline and effectively apply design principles during project creation demanded accelerated expertise development (Popovic, 2004) supported by the course’s resource-based learning design (Hill & Hannafin, 2001).
The practices, processes, and design journeys identified in our analysis allow us to identify recommendations for course design that can support learners throughout the creation of complex e-learning projects while developing technical expertise in e-learning authoring:
I) Include opportunities for learners to discuss their design process, externalizing design judgments. In this study, participants with some prior experience in e-learning development provided a higher level of conscious explanation of decisions influenced by design judgments. Complete novices, such as Beth and Robert in this study, benefited from the opportunity to discuss their project progress during development and provide a reflection upon project completion. These reflections, guided by instructor-provided prompts (Fessl et al., 2017), can inform future success in practice (Elvira et al., 2017). As participants recollect crucial design judgments, rules of thumb can be established (Ertmer et al., 2009). The externalization of design decisions and obstacles can also aid instructions in identifying the need to provide just-in-time guidance (Stefaniak et al., 2022). Instructors should provide opportunities for students to externalize reflection at the beginning, middle, and end of project creation so valuable feedback can be provided for specific issues that arise in different phases of the process.
i) Provide guidance and activities for problem-solving complex framing and development steps: Students can experience difficulties that arise from, for example, project scope, time constraints, and tool functionality during project development. In the two semesters in which data was collected, the instructor was available to address students’ concerns and technical questions. Our analysis suggests, however, struggles which pertain mainly to project scope and complex development steps could be resolved with more comprehensive guidance included in the online course itself. Our recommendation is that instructors offer specific examples of project scope and learning objectives in addition to opportunities to learn common in-tool procedures, such as accessibility implementation in worked examples (Elvira et al., 2017), before learners are required to perform these tasks for their complex projects. This approach might remove the time-consuming scoping and problem-solving we identified via framing and navigational judgments coded.
This study aimed to capture a chronology of design judgments (Lachheb & Boling, 2021; Nelson & Stolterman, 2012) enacted by six novice e-learning designers in a 4-week experience focusing on developing an interactive e-learning object. Our second-order narrative (Creswell & Poth, 2023) allowed us to capture the evolution of design judgments made by participants and reveal the existence of experiential and contextual variables (Stefaniak, 2023) that added complexity to participants’ design processes. Based on our findings, we identified the need for a high level of instructor involvement, particularly as novice IDs establish the constraints of their projects and learn how to use development tools. Our results also illustrate that recognizing one’s design judgments and reflecting upon design and development practices can be important in developing e-learning expertise. Incorporating reflection into instruction can assist learners in developing problem-solving strategies for e-learning project creation that are crucial in the evolving L&D landscape.
This study presents, however, some limitations. As we set forth to focus on the complexity of participants’ judgments, the main limitation of this study relates to the small number of participants whose data we collected and analyzed. Our analysis does not allow us to make generalized conclusions that apply to all populations. Our conclusions should be viewed as a context-dependent humble theory (Cobb et al., 2003) since our participant demographic was centered on learners between the ages of 30 and 45 and was primarily Caucasian. Follow-up studies might focus on the experiences of traditional college-age students of diverse backgrounds to identify other variables that might influence design processes for different populations.
Furthermore, our study focused only on data from an asynchronous online course. Subsequent research focusing on design processes in face-to-face courses could provide valuable insights into, for instance, emotions that may influence design decisions and design judgments, as well as how IDs’ personalities can factor in the design process. Finally, another limitation of this study relates to its individual project nature. Since participants designed and developed these projects independently, our findings might not apply to collective project creation experiences in which participants collaborate with peers and subject-matter experts. However, our study contributes to a growing understanding of how IDs and e-learning developers of distinct experience levels enact design judgments.
EdTech Archives