Leveraging Learning Engineering to Advance Authentic Learning in Virtual Reality: Applications in Immersive Learning Design

Introduction

Virtual reality (VR) is an increasingly popular medium for delivering authentic learning experiences, but widely adopted frameworks and an established set of agreed-upon best practices for immersive learning experience design have yet to be defined (LaRocco, 2020; Wang et al., 2021). Thus, it is up to individual creators and teams to experiment, investigate, and formulate approaches and frameworks to create a set of best practices and standards which foster authentic learning and promote learning outcomes. Learning engineering provides a viable approach to address the lack of commonly accepted VR learning design practices for two reasons:

1. Learning engineering provides a set of evidence-based principles and practices (Goodell, Kolodner, et al., 2022) from which VR learning designers can draw to create authentic, immersive learning solutions.
2. Learning engineering emphasizes investigating and validating effective and novel learning design methodologies which are scalable and replicable (Goodell & Kolodner, 2022).

A learning engineering approach to VR design can advance the immersive learning field. This methodology makes authentic learning accessible and maximizes the unique benefits of three-dimensional, multisensory, interactive, experiential environments. By examining how learning engineering practices can be leveraged in VR learning design to develop authentic and impactful learning experiences, this position paper aims to contribute to the ongoing discourse of harnessing emerging technologies to achieve greater learning outcomes and more broadly advance the future of immersive learning design.

This paper begins by defining the learning engineering process, followed by an exploration of its primary components. The subsequent sections offer practices, tools, and example cases to demonstrate their application in designing and developing VR learning experiences based on learning engineering principles.

Learning Engineering as an Approach to VR Learning Design

The IEEE International Consortium for Innovation and Collaboration in Learning Engineering (ICICLE) defines learning engineering as: “a process and practice that applies the learning sciences using human-centered engineering design methodologies and data-informed decision-making to support learners and their development” (Goodell, 2022, p.10). Learning engineering is a repeatable process intended to iteratively create, implement, and investigate learning solutions, as defined in the Learning Engineering Toolkit (Figure 1) (Kessler et al., 2022). It applies competencies from many domains, including learning science, computer science, data science, learning experience design, subject-matter expertise, assessment, measurement, and evaluation, learning environment engineering, education, and training, and is often accomplished in multidisciplinary teams (Goodell, 2022; Wagner et al., 2018). While learning engineering practices draws from these established disciplines, its process is unique. It exhibits concurrent consideration and cycles of creation, implementation, and investigation factors to address a challenge (Kessler et al., 2022).

Learning engineering always begins with a problem or a challenge that needs to be addressed—typically an opportunity to improve learning or the conditions for learning (Kessler et al., 2022). Just as engineering is the practice of using natural science to solve technical problems and improve systems, learning engineering is the practice of using learning science to solve learning problems and improve learning conditions (Goodell & Thai, 2020; Kessler et al., 2022; Kolodner, 2023).

It's important to note that learning engineering is not specific to immersive learning. It can be implemented in the design and iterative development of all types of learning environments, whether technologically mediated or not.

Figure 1

Learning Engineering Process CC BY Aaron Kessler, Jim Goodell, Sae Schatz (Goodell & Kolodner, 2022)

Understanding the challenge involves getting a clear picture of the problem that the VR learning solution seeks to address, the learning objectives, the learner demographic, and the conditions that will hinder or help learners achieve the learning goal. A project may face multiple challenges, such as business, design, implementation, and instrumentation. For example, a design challenge might involve identifying the correct target behaviors for a task to recreate these practice opportunities in VR. The learning engineering process may look like this:

• Challenge: Identify a collection of key behaviors, related errors, and misunderstandings.
• Creation: This challenge can be clarified by creating interview questions for subject matter experts (SME) that elicit knowledge and behaviors or successful performance.
• Implementation: SME interview questions can be implemented with multiple SMEs, and data can be captured and organized.
• Investigation: Analysis of responses can reveal themes, enabling the identification of the key behaviors.

These key behaviors inform the following design challenges, such as creating the VR solution. A sub-iterative design process follows the investigation, creation, and user testing cycle within the creation phase. Imagine the design team has conceptualized a task for the VR experience and wants to validate whether it is on the right track. The sub-iterative design process may look like:

• Creation: Development of a low-fidelity prototype depicting the task within the VR environment.
• User testing: Recruiting a small group of users to playtest the prototype.
• Investigation: Collecting user experience data through a survey and debrief session.

Findings from the investigation would then be used to inform the iterative design, circling back to the creation phase.

Learning engineering is iterative and data-driven. Each process component helps to inform the others, with the challenge sitting at the heart of the process and data-informed decisions as the catalyst to engineer solutions from point A to point B (Kessler et al., 2022).

To further unpack learning engineering as “a process and practice that applies the learning sciences using human-centered engineering design methodologies, and data-informed decision-making to support learners and their development” (Goodell, 2022, p.10), the following sections detail learning sciences, human-centered engineering design, and data-informed decision-making for VR learning design more in-depth.

Learning Sciences and Learning Design Principles for VR Design

Learning sciences can provide an initial blueprint for designing authentic learning experiences. Considering what is known to be true about how people learn facilitates the pedagogy, selection, and application of learning design principles that lay the foundation for effective design (Goodell, Kolodner, et al., 2022; McEldoon, 2023). Anchoring the design of VR learning solutions in the learning sciences enables the implementation of research-validated strategies that maximize design effectiveness and increase the likelihood of producing desired learning outcomes.

VR learning experiences are a powerful method of modeling systems by immersing the learners within real-life settings. This approach provides an optimal environment for situated cognition and experiential learning, allowing learners to actively participate in the learning process (Aiello et al., 2012; Choi & Hannafin, 1995; Lave & Wenger, 1991). By interacting with the virtual world, learners can transform their experiences into cognitive understanding (Aiello et al., 2012; Choi & Hannafin, 1995; Lave & Wenger, 1991). This type of immersion, which encourages solving real-world problems similar to those encountered in the physical world, promotes authentic learning (Herrington & Oliver, 2000).

When grounding practice in authentic learning environments, it is optimal for the constraints of the tasks to closely resemble real-world tasks for learners to suspend disbelief and accept the environment as realistic (McEldoon, 2023; Wang et al., 2021). It is encouraged that VR learning scenarios attempt to effectively replicate situational complexity and contextual cues that learners must attend to in the physical world when performing the same tasks to promote the possibility of repeated practice, better-equipping learners to transfer these skills outside of the experience (Wang et al., 2021). Doing so increases scenario fidelity, the degree to which rules, behaviors, and environment properties are reproduced in a simulation compared to the real world (Ragan et al., 2015). Scenario fidelity contributes to the level of presence and immersion the VR system imparts to learners (Ragan et al., 2015).

Immersion

Immersion makes a VR experience feel real and believable and is a primary affordance of learning in VR (Jerald, 2015; Pagano, 2013). Nilsson, Nordahl, and Serafin (2016) suggest that there are three-dimensional categories of immersion. Table 1 outlines each immersion category and recommendations for how a VR experience can prompt them.

Table 1

Three-dimensional categories of immersion and how they can be promoted

Flow, Desirable Difficulty, and Autonomy

Flow is a state in which an individual is deeply focused, channeling their emotions and attention into an activity they are engaged in (Csikszentmihalyi, 1990; Villanueva & Vaidya, 2019; Webster et al., 1993). As the ZPD and flow state is achieved through a careful balance of challenge and skill level (Csikszentmihalyi, 1990; McEldoon, 2023; Webster et al., 1993), immersive learning designers should give careful consideration to designing for desirable difficulty within their VR experiences—challenging, but not too difficult. Imparting learners with the freedom and control to interact with a sense of agency increases motivation and precipitates flow behavior conducive to learning (McEldoon, 2023; Villanueva & Vaidya, 2019). VR experiences prioritizing a sense of autonomy enable learners to experience cause-and-effect relationships firsthand (Torrence, 2022). Learners can uncover the interrelationships between the concepts and their actions through active experimentation with decision-making, testing of solutions, receiving feedback, and interpreting and internalizing their experience (Aiello et al., 2012; Villanueva & Vaidya, 2019). As learners interact within the virtual world, they construct new ideas and build upon their prior knowledge (Aiello et al., 2012).

Repetition and Active Learning

Repeated and varied practice in immersive, simulated environments offers learners familiarity and confidence with target situations through normalizing exposure to authentic scenarios (Fahl et al., 2023; Sirakaya & Kilic Cakmak, 2018). Promoting active learning by presenting opportunities for learners to engage in the virtual world and construct their understanding is essential for cultivating authentic and effective immersive learning environments and even promotes a sense of presence (Aiello et al., 2012; Bowman & McMahan, 2007; Chen & Wang, 2018).

Presence

Presence is a person’s subjective experience when interacting within VR environments (Bowman & McMahan, 2007; Nilsson et al., 2016; Slater, 2018). It is the feeling of being in an environment when physically located elsewhere. A learner’s sense of presence is stronger with interactivity, which helps them buy into the illusion that they are there (Mütterlein, 2018). Because presence is integral to how people learn through immersive modalities, any design decisions that break presence risk jeopardizing the learning outcomes of the experience. Thus, maintaining presence is of the utmost importance in immersive learning design.

Managing Cognitive Load

For VR learning experiences to be effective, it’s important that they are designed to direct learner attention to target information related to a task and to limit distractions that can interfere with their ability to do the task (Kalyuga et al., 1999; Makransky & Petersen, 2021; Mayer, 2005; Sweller, 2011). Specifically, it is essential to consider learners’ information processing limitations in designs to minimize extraneous cognitive load (McEldoon, 2023).

Presenting excessive stimuli across multiple modalities in an immersive experience can overwhelm learners’ information-processing systems, and this difficulty in parallel processing can adversely affect learning outcomes (Anderson & Crawford, 1980; Kalyuga et al., 1999; Mayer et al., 2001; Moreno & Mayer, 2000). Therefore, designers of immersive learning experiences should use caution and refrain from stimulating concurrent competing modalities, which can reduce cognitive resources allocated toward integration, memory, and transfer, thereby interfering with the learning of target material. For instance, if a learner makes a mistake performing a task in VR, there are three primary senses through which the system can send signals to the learner—visual stimuli by way of user interface text or images, auditory through sound effects or voice-over audio, or somatosensory through haptics. The likelihood of competition if stimulating all three senses is high; however, several two-stimulus permutations may work synergistically if the intensity is carefully balanced to not compete with one another and overwhelm users. The following design case exemplifies how learning sciences can be considered and translated into design decisions for VR training.

Example Design Case: HVAC Training

Designers were tasked to create a fully immersive VR learning experience teaching heating, ventilation, and air conditioning (HVAC) procedures. They practiced learning engineering and applied the learning sciences to guide their initial design decisions (McEldoon & Torrence, 2023). The learning designers prioritized the following learning design principles based on the learning sciences: focused attention, minimizing extraneous cognitive load, formative feedback, and evidence-based performance measurement. Table 2 details the design decisions and artifacts created to address the management of learner attention and cognitive load, feedback mechanisms, and the application of the Evidence-Centered Design for Assessment (ECD) framework.

Table 2

Design decisions by learning science consideration for the HVAC VR case (McEldoon & Torrence, 2023)

Feedback was scaffolded to nudge learners to adjust their behavior at earlier instances of incorrect actions, with the degree of corrective feedback increasing as learners’ misunderstandings became more evident. This approach also effectively manages cognitive load because providing voice-over corrective feedback for every learner mistake could overwhelm learners and adversely affect learning.

Evidence-Centered Design for Assessment (ECD) is a conceptual framework centered on constructing evidence-based claims grounded in empirical data (Mislevy et al., 2003; Torrence, 2023a). The HVAC training used ECD to tackle the challenge of seamlessly weaving valid and reliable measurements into the VR experience (McEldoon & Torrence, 2023). ECD is particularly valuable to VR modalities because assessment can be embedded directly within the experience without breaking presence and immersion or disrupting flow (Shute et al., 2017; Torrence, 2022a). By applying ECD to the design of the HVAC VR experience, strategic alignment was achieved between the learning tasks designed and the target competencies (Torrence, 2022a).

Operationalizing learning objectives and competencies into key demonstrable behaviors, which serve as empirical evidence of competency levels, is a systematic approach to design (Shute et al., 2017; Torrence, 2022). Learning designers can use these behaviors as design requirements when constructing VR learning tasks (Shute et al., 2017). This practice ensures that learners' behaviors are authentic to the real-life task and that performance measured in the VR experience accurately represents learner competency levels (Mislevy et al., 2003; Parra et al., 2021; Shute et al., 2017). Practitioners can use data collected from learner interactions to make evidence-based inferences to guide pedagogical decision-making about the learning journey and provide useful information to learners in formative feedback cycles (Torrence, 2023a).

When adopting a learning engineering approach, immersive learning practitioners may want to identify the learning design principles most applicable to their learning solution. These principles can then be defined as design pillars, serving as alignment criteria for instructional design decisions.

Human-Centered Engineering VR Design

Learning engineering is a process and practice that uses human-centered engineering design methodologies to design from the perspective of the end-users of the solution (Goodell, 2022; Thai et al., 2022). This section will spotlight learner-centered design, participatory design, and prototyping methodologies from human-centered and engineering disciplines for VR learning design and provide recommendations for practice.

Learner-Centered Design

Learner-centered design is primarily concerned with learners as the end-user and the efficacy of their learning (Goodell & Kolodner, 2022; Norman & Spohrer, 1996; Schiller, 2009). Persona development is critical for making VR designs resonate with the target learners (Torrence & Mueller, 2022, pp. 29-34). Personas are design tools that represent hypothetical archetypes of individuals (Schatz et al., 2022; Thai et al., 2022). Learner and character personas are two popular types applicable to VR learning solutions. The information described in personas can vary, but learner personas commonly detail a single archetype’s demographics, biography, prior knowledge, goals, motivations, fears, values, perspectives, and notable traits (Torrence & Mueller, 2022). Learning designers can then use this information to shape the context, storyline, appropriate content, and difficulty level of the VR experience. The goal is to develop an array of learner personas that capture the breadth and depth of the target learners and their diverse perspectives, preferences, and dispositions so that the personas can be used as an accountability tool for validating whether the designs meet learners’ needs.

Figure 2

Learner Persona Example (Torrence & Mueller, 2022, p. 30)

Character personas are essential for VR learning solutions that feature non-player characters (NPCs) or virtual humans (VHs). They contribute to developing realistic character interactions by providing depth to fears, motivations, and flaws, defining emotional ranges, and ensuring factual consistency in storytelling. Learning designers can use a character’s personality traits to strategically plan the emotional tenor of a conversation and how the learner’s behavior will impact the character emotionally (Torrence & Mueller, 2022).

Practitioners can conduct interviews, host focus groups, administer surveys, derive information from system data, and analyze trends from these data sources to construct initial drafts of personas. A validation step with target learners or periphery personnel should occur to verify that the assumptions made in the personas are accurate and iterate if necessary.

Participatory Design

Participatory design is a collaborative approach involving end-users in the entire design process (B.-N.Sanders, 2002; Thai et al., 2022). By engaging high-interest and high-influence stakeholders in the participatory design process and learning about their experiences, learning designers can use these findings as a source of inspiration and ideation for enhanced learning design and create a more effective and authentic VR learning solution (B.-N.Sanders, 2002; Mueller & Torrence, 2023). Participatory design typically involves a series of activities where participants self-express in the creation of artifacts contributing to the design (Schatz et al., 2022), which presumably facilitates the creation of a VR learning solution that resonates more with the target audience due to the diverse array of people with different identities that have contributed to the end product.

Prototyping

Prototypes are an essential human-centered engineering design tool for VR learning designers (Goodell & Kolodner, 2022). Prototyping aims to depict ideas concretely, without concern for aesthetics or perfection, to solicit actionable feedback or validate the design (Jerald, 2015). Given a VR learning solution's long and intensive development lifecycle, prototyping helps affirm that design efforts meet the requirements and stakeholder expectations before expending many development hours to execute the design. Various prototypes exist, ranging from low-fidelity paper models to high-fidelity functional versions (Barr et al., 2022; Jerald, 2015; Schatz et al., 2022). The choice of prototype could be influenced by how easily the VR design can be communicated through the selected medium, as well as the familiarity and experience of the user testers and stakeholders offering critique with that medium. A storyboard might communicate the look and feel to a client with little VR experience, while a block diagram may communicate progression to a development team. Example storyboard and block diagram prototypes are further detailed in the following sections.

Storyboard

Storyboards typically convey scene-by-scene depictions of what learners will experience in VR from a first-person point of view. Storyboard illustrations give a general sense of the interface, positioning of objects, and interactivity. Effectively communicating the immersive design is the ultimate purpose of storyboarding. However, storyboards do not need to be high-fidelity to convey a design effectively or use advanced interaction design software. Figure 3 depicts a storyboard scene for a VR learning experience intended to develop proficiency in operating a micrometer. The storyboard was created with standard presentation software, using text, shapes, and royalty-free images to convey an immersive design (Torrence, 2023b).

Storyboards are also an avenue to communicate design decisions that result from constraints, limitations, or conditions that must be satisfied. For instance, one constraint of the micrometer VR experience is the inability to use locomotion to reduce the likelihood of motion sickness. The storyboard snapshot in Figure 3 illustrates how a learner would use their controller to teleport to a location (Torrence, 2023b).

Figure 3

Storyboard Example

Note. The storyboard scene for the micrometer VR learning experience illustrates teleportation using a controller (Torrence, 2023b).

Creating storyboards facilitates better communication of the design with the software development team, project stakeholders, and end-users. Presenting storyboards to target learners early in the design process is advantageous for gathering feedback on the scenario's authenticity and the task design's fidelity. This feedback should be collected before proceeding with development to evaluate whether the design meets learners' needs and to make necessary iterations if it falls short.

Block Diagram

Another method of conveying the design of a VR learning experience is to illustrate it using a block diagram. A block diagram conveys a high-level view of the system components and their interconnections (Jerald, 2015). Figure 4 conveys how learners will progress through Level 1 of the micrometer operations VR experience.

Figure 4

Block Diagram Example

Note. The block diagram depicts a subsection of Level 1 of the VR learning experience teaching micrometer operations. The same sequence of behaviors is required for Gate-02 and Gate-03, starting with “Learner picks up micrometer” (Torrence, 2023b).

A benefit of depicting experience progression using a block diagram is the ease with which design patterns can be detected in the experience. Design patterns are reusable conceptual solutions to commonly occurring problems (Barr et al., 2022; Goodell, Hampton, et al., 2022; Jerald, 2015). Notice in Figure 4 that after learners have navigated to an item, the process requires the same sequence of behaviors before they are considered to have completed the micrometer measurement of that item. Any task in which learners are expected to measure an item with a micrometer follows this same sequence of requirements. This same pattern is prevalent in subsequent levels of this VR experience, where learners must measure an item with a micrometer, whether the measurement is a practice attempt or a tracked observation (Figure 5). Identifying design patterns can help expedite the development of these elements, produce consistency in the experience, and facilitate the fast and standardized design of similar components in the future (Goodell & Kolodner, 2022; Jerald, 2015).

Figure 5

Block Diagram Example 2

Note. The block diagram depicts a subsection of Level 2 of the VR learning experience teaching micrometer operations. The same sequence of behaviors from Figure 4 also appears in Gate-04, Gate-05, and Gate-06 (Torrence, 2023b).

Data-Informed Decision-Making for VR

Learning engineering is data-driven (Goodell & Kolodner, 2022). Every part of the learning engineering process utilizes data-informed decision-making (Czerwinski et al., 2022). When attempting to understand the challenge, there is naturally uncertainty around this problem that affects decisions that need to be made (Goodell & Kolodner, 2022; Hubbard, 2014). The purpose of measurement is to gather meaningful data to support the decision-making process, and this evidence can be used to guide the iterative creation, implementation, and investigation processes.

Creation Phase

Data-informed decision-making in the creation phase might begin with investigation through interviews or focus groups with stakeholders, subject-matter experts, and target learners to learn more about the challenge to address with the VR design. Outputs of such efforts could yield the creation of personas and performance task analysis results. Immersive learning designers can use this data to inform design decisions about creating the VR learning experience. Once a design has been conceptualized and documented, learning designers can conduct end-user testing to collect feedback for further iteration. This might look like first providing design documentation or storyboards in front of learners to gather their thoughts and revising the design based on the feedback received to better fit the needs of learners. This iterative design cycle of creation, user testing, and investigation continues from the prototype phase until the creation of the VR learning solution.

End-user testing is essential during the creation phase because design is rooted in assumptions about what will work. These assumptions need to be validated. Table 3 describes examples of assumptions practitioners may have when designing and developing a VR learning solution and the data collection methods they may pursue to investigate and validate their assumptions.

Table 3

Common design assumptions during the learning engineering creation phase of a VR project and example data collection methods

Within-VR measures can also be used to investigate research questions relating to a learning solution. Game-based learning solutions built in Unity commonly use Experience API (xAPI) to collect data about learners’ behaviors as they play (Almond, 2019; Kim et al., 2016). Many VR applications are built using the Unity game engine and can utilize these same standards and libraries to track learners’ behavior in VR (Dig-iT! Games LLC, 2021; Yet Analytics, Inc., 2023). Implementing data standards, such as xAPI, in VR learning solution development, can enable stealth assessment of real-time performance (Shute et al., 2017) and allow adaptive VR experiences. Additionally, virtual choreographies (Cassola et al., 2022) support design and performance measurement, which is beneficial for developing flexible VR designs. As detailed earlier in this paper, ECD approaches encourage preparatory efforts to identify and operationalize key behaviors for VR learning solutions. Identifying these target indicators helps define the system triggers needed to create xAPI statements (Mislevy, 2013; Shute et al., 2017; Torrence, 2023a). Practitioners are encouraged to collaborate with domain experts, psychometricians, and software designers early in the design process so that the system is designed to collect the right data to answer the research questions relating to the VR solution (Mislevy, 2013).

Implementation & Investigation Phases

Once a VR learning solution has been created, practitioners must face different assumptions to investigate. Common assumptions during implementation are that the VR learning experience produces positive learning outcomes— the solution equips learners with task and decision-making competence, and learners can transfer what they have learned on the job, positively impacting business outcomes (Thalheimer, 2018; Torrence, 2023c). Other assumptions include increased confidence levels, perceptions, intentions, and motivations related to the subject matter due to the VR training. Additionally, it is assumed that the theory of change regarding learners and situational factors is valid, and that the conditions implementing the VR solution are favorable (Torrence, 2023c). Similarly, these assumptions need to be evaluated through instrumentation and analysis.

The investigation should yield answers regarding the efficacy of the VR solution design and implementation conditions, which will either support that certain design decisions are effective or ineffective or require more data to be able to conclude. Findings should be used to inform other parts of the learning engineering cycle (Goodell & Kolodner, 2022). Effective elements should be encouraged in future designs, while ineffective elements should undergo iterative design, and efforts to collect more meaningful data should be planned. The cyclical process of learning engineering is oriented toward continually improving the learning experience and is informed by results over time (Goodell & Kolodner, 2022).

Learners are not static, nor are implementation conditions nor the competencies being taught. Continuous data collection and data-informed decision-making about VR solutions are vital. The more data immersive learning designers collect over time relating to VR learning solutions, the more can be learned about effective learning design for VR. Their understanding of their learner population and implementation contexts will be more robust. Replication trends in the data will reveal and validate practices that work or build on findings and present new approaches to investigate until best practices for VR learning design are identified.

Conclusion

This position paper advocates for the application of learning engineering processes and practices for the design of authentic VR learning experiences. Widespread awareness and adoption of learning engineering processes and practices as an approach to VR learning experience design can revolutionize how immersive learning solutions are created and set a new standard of excellence for virtual reality design and development.

Learning engineering’s emphasis on applying the learning sciences as a fundamental step in establishing design requirements reinforces the importance of evidence-informed selection of learning design principles. This helps to ensure that VR learning experiences align with how people learn (Goodell & Kolodner, 2022; McEldoon, 2023). The utilization of human-centered engineering design methodologies such as learner-centered design, participatory design (Elizarova & Dowd, 2017; Goodell & Kolodner, 2022; Mueller & Torrence, 2023), and tools such as personas (Torrence & Mueller, 2022), storyboards, and block diagrams (Torrence, 2023d) help to facilitate the creation of authentic and effective VR learning experiences for end-users. Data-informed decision-making propels the learning engineering process, supporting learners and their development at scales ranging from individual VR learning tasks to VR programs, to even entire learning ecosystems. Data-driven design, development, and continuous improvement of authentic, immersive learning solutions can lead to optimized instructional design approaches, efficient iterative development cycles, improved production systems, and curated implementation contexts. These enhancements favor positive learning outcomes and promote behavior change over time.

Learning engineering processes and practices can aid in discovering and identifying effective VR learning design approaches. They also support investigations into why these approaches work, using data-driven efforts to establish and scale best practices for creating authentic, immersive learning experiences. It is imperative that the immersive learning industry establish best practices for designing effective and authentic VR learning solutions, and learning engineering can aid in investigating and advancing the understanding of these standards.

References

1. Aiello, P., D’Elia, F., Di Tore, S., & Sibilio, M. (2012). A constructivist approach to virtual reality for experiential learning. E-Learning and Digital Media, 9(3), 317–324. https://doi.org/10.2304/elea.2012.9.3.317
2. Almond, R. G. (2019). A four-process implementation of game-based scoring. Retrieved from http://gbaworkshop.tntlab.org/wp-content/uploads/2019/08/Education-2-Almond.pdf
3. Anderson, J. R., & Crawford, J. (1980). Cognitive psychology and its implications. San Francisco: W.H. Freeman.
4. Barr, A., Dargue, B., Goodell, J., & Redd, B. (2022). Learning engineering is engineering. In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 125–152). Routledge.
5. B.-N.Sanders, E. (2002). From user-centered to participatory design approaches. In J. Frascara (Ed.), Design and the Social Sciences (Vol. 20020425, pp. 1–8). CRC Press. https://doi.org/10.1201/9780203301302.ch1
6. Bowman, D. A., & McMahan, R. P. (2007). Virtual reality: How much immersion is enough? Computer, 40(7), 36–43. https://doi.org/10.1109/MC.2007.257
7. Brooke, J. (1996). SUS: A ‘quick and dirty’ usability scale. In P.W. Jordan, B. Thomas, B.A. Weerdmeester, & I.L. McClelland (Eds.), Usability Evaluation in Industry. London: Taylor and Francis.
8. Cassola, F., Mendes, D., Pinto, M., Morgado, L., Costa, S., Anjos, L., Marques, D., Rosa, F., Maia, A., Tavares, H., Coelho, A., & Paredes, H. (2022). Design and evaluation of a choreography-based virtual reality authoring tool for experiential learning in industrial training. IEEE Transactions on Learning Technologies, 15(5), 526–539. https://doi.org/10.1109/TLT.2022.3157065
9. Chen, Y.-H., & Wang, C.-H. (2018). Learner presence, perception, and learning achievements in augmented–reality–mediated learning environments. Interactive Learning Environments, 26(5), 695–708. https://doi.org/10.1080/10494820.2017.1399148
10. Choi, J.-I., & Hannafin, M. (1995). Situated cognition and learning environments: Roles, structures, and implications for design. Educational Technology Research and Development, 43(2), 52–69.
11. Csikszentmihalyi, M. (1990). Literacy and intrinsic motivation. Daedalus, 119(2), 115–140. https://www.jstor.org/stable/20025303 
12. Czerwinski, E., Goodell, J., Ritter, S., Sottilare, R., Thai, K.-P., & Jacobs, D. (2022). Learning engineering uses data (part 1): Instrumentation. In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 153–174). Routledge.
13. Dig-iT! Games LLC. (2021). UnityGBLxAPI. Retrieved from https://github.com/gblxapi/UnityGBLxAPI
14. Elizarova, O., & Dowd, K. (2017, December 14). Participatory Design in Practice. UX Magazine. Retrieved from https://uxmag.com/articles/participatory-design-in-practice
15. Fahl, J. T., Duvivier, R., Reinke, L., Pierie, J.-P. E. N., & Schönrock-Adema, J. (2023). Towards best practice in developing motor skills: A systematic review on spacing in VR simulator-based psychomotor training for surgical novices. BMC Medical Education, 23(1), 154. https://doi.org/10.1186/s12909-023-04046-1
16. Goodell, J. (2022). Introduction: What is learning engineering? In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 5–25). Routledge.
17. Goodell, J., Hampton, A. J., Tong, R., & Schatz, S. (2022). Software and technology standards as tools. In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 311–331). Routledge.
18. Goodell, J., & Kolodner, J. (Eds.). (2022). Learning engineering toolkit: Evidence-based practices from the learning sciences, instructional design, and beyond (1st ed.). Routledge. https://doi.org/10.4324/9781003276579
19. Goodell, J., Kolodner, J., & Kessler, A. (2022). Learning engineering applies the learning sciences. In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 47–82). Routledge.
20. Goodell, J., & Thai, K.-P. (2020). A learning engineering model for learner-centered adaptive systems. In HCI International 2020–Late Breaking Papers: Cognition, Learning and Games: 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings 22. Springer International Publishing, 557–573. https://doi.org/10.1007/978-3-030-60128-7_41
22. Jerald, J. (2015). The VR book: Human-centered design for virtual reality. Morgan & Claypool.
23. Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13(4), 351–371. https://doi.org/10.1002/(SICI)1099-0720(199908)13:4<351::AID-ACP589>3.0.CO;2-6
24. Kessler, A., Craig, S. D., Goodell, J., Kurzweil, D., & Greenwald, S. W. (2022). Learning engineering is a process. In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 29–46). Routledge.
25. Kim, Y. J., Almond, R. G., & Shute, V. J. (2016). Applying evidence-centered design for the development of game-based assessments in physics playground. International Journal of Testing, 16(2), 142–163. https://doi.org/10.1080/15305058.2015.1108322
26. Klepsch, M., Schmitz, F., & Seufert, T. (2017). Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load. Frontiers in Psychology, 8, 1997. https://doi.org/10.3389/fpsyg.2017.01997
27. Kolodner, J. L. (2023). Learning engineering: What it is, why I’m involved, and why I think more of you should be. Journal of the Learning Sciences, 32(2), 305–323. https://doi.org/10.1080/10508406.2023.2190717
28. LaRocco, M. (2020). Developing the ‘best practices’ of virtual reality design: Industry standards at the frontier of emerging media. Journal of Visual Culture, 19(1), 96–111. https://doi.org/10.1177/1470412920906255
29. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.
30. Makransky, G., & Lilleholt, L. (2018). A structural equation modeling investigation of the emotional value of immersive virtual reality in education. Educational Technology Research and Development, 66(5), 1141–1164. https://doi.org/10.1007/s11423-018-9581-2
31. Makransky, G., & Petersen, G. B. (2021). The cognitive affective model of immersive learning (CAMIL): A theoretical research-based model of learning in immersive virtual reality. Educational Psychology Review, 33(3), 937–958. https://doi.org/10.1007/s10648-020-09586-2
32. Mayer, R. E. (Ed.). (2005). The Cambridge handbook of multimedia learning. Cambridge University Press.
33. Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93(1), 187–198. https://doi.org/10.1037/0022-0663.93.1.187
34. McEldoon, K (Ed.). (2023). The learning design principles: A toolkit for evidence-based learning design. Retrieved from https://www.pearson.com/en-gb/efficacy/learning-evidence.html#oldp
35. McEldoon, K., & Torrence, K. (2023, July 24). Improving VR-based workforce learning design with the learning design principles: A talespin case study. IEEE ICICLE Learning Engineering Conference, Pittsburgh, PA, USA.
36. McMahan, R. P., Bowman, D. A., Zielinski, D. J., & Brady, R. B. (2012). Evaluating display fidelity and interaction fidelity in a virtual reality game. IEEE Transactions on Visualization and Computer Graphics, 18(4), 626–633. https://doi.org/10.1109/TVCG.2012.43
37. Mislevy, R. J. (2013). Evidence-centered design for simulation-based assessment. Military Medicine, 178(10S), 107–114. https://doi.org/10.7205/MILMED-D-13-00213
38. Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29. https://doi.org/10.1002/j.2333-8504.2003.tb01908.x
39. Moreno, R., & Mayer, R. E. (2000). A coherence effect in multimedia learning: The case for minimizing irrelevant sounds in the design of multimedia instructional messages. Journal of Educational Psychology, 92(1), 117. https://doi.org/10.1037/0022-0663.92.1.117
40. Mueller, R., & Torrence, K. (2023, October 26). Participatory design: A collaborative approach to VR training. DevLearn Conference & Expo, Las Vegas, NV, USA.
41. Mütterlein, J. (2018, January). The three pillars of virtual reality? Investigating the roles of immersion, presence, and interactivity. 51st Hawaii International Conference on System Sciences (HICSS), Big Island, HI, USA. https://doi.org/10.24251/HICSS.2018.174
42. Nilsson, N. C., Nordahl, R., & Serafin, S. (2016). Immersion Revisited: A review of existing definitions of immersion and their relation to different theories of presence. Human Technology, 12(2), 108–134. https://doi.org/10.17011/ht/urn.201611174652
43. Norman, D. A., & Spohrer, J. C. (1996). Learner-centered education. Communications of the ACM, 39(4), 24--27. https://doi.org/10.1145/227210.22721
44. Pagano, K. O. (2013). Immersive learning: Designing for authentic practice. Association for Talent Development.
45. Parra, E., Chicchi Giglioli, I. A., Philip, J., Carrasco-Ribelles, L. A., Marín-Morales, J., & Alcañiz Raya, M. (2021). Combining virtual reality and organizational neuroscience for leadership assessment. Applied Sciences, 11(13), 5956. https://doi.org/10.3390/app11135956
46. Ragan, E. D., Bowman, D. A., Kopper, R., Stinson, C., Scerbo, S., & McMahan, R. P. (2015). Effects of field of view and visual realism on virtual reality training effectiveness for a visual scanning task. IEEE Transactions on Visualization and Computer Graphics, 21(7), 794–807. https://doi.org/10.1109/TVCG.2015.2403312
47. Schatz, S., Thai, K.-P., Craig, S. D., Schoenherr, J. R., Lis, J., & Kolodner, J. (2022). Human-centered design tools. In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 279–301). Routledge.
48. Schiller, S. Z. (2009). Practicing learner-centered teaching: Pedagogical design and assessment of a second life project. Journal of Information Systems Education, 20(3), 369.
49. Shute, V., Rahimi, S., & Emihovich, B. (2017). Assessment for learning in immersive environments. In D. Liu, C. Dede, R. Huang, & J. Richards (Eds.), Virtual, Augmented, and Mixed Realities in Education (pp. 71–87). Springer Singapore. https://doi.org/10.1007/978-981-10-5490-7_5
50. Sirakaya, M., & Kilic Cakmak, E. (2018). Effects of augmented reality on student achievement and self-efficacy in vocational education and training. International Journal for Research in Vocational Education and Training, 5(1), 1–18. https://doi.org/10.13152/IJRVET.5.1.1
51. Slater, M. (2018). Immersion and the illusion of presence in virtual reality. British Journal of Psychology, 109(3), 431–433. https://doi.org/10.1111/bjop.12305
52. Sutcliffe, A., Gault, B., & Shin, J.-E. (2005). Presence, memory and interaction in virtual environments. International Journal of Human-Computer Studies, 62(3), 307–327. https://doi.org/10.1016/j.ijhcs.2004.11.010
53. Sweller, J. (2011). Cognitive load theory. In J.P. Mestre & B.H. Ross (Eds.), Psychology of Learning and Motivation, 55 (pp. 37-76). Academic Press. https://doi.org/10.1016/B978-0-12-387691-1.00002-8
54. Thai, K.-P., Craig, S. D., Goodell, J., Lis, J., Schoenherr, J. R., & Kolodner, J. (2022). Learning engineering is human-centered. In J. Goodell & J. Kolodner (Eds.), Learning Engineering Toolkit (pp. 83–123). Routledge.
55. Thalheimer, W. (2018).The learning-transfer evaluation model: Sending messages to enable learning effectiveness. Retrieved from: https://WorkLearning.com/Catalog
56. Torrence, K. (2022, October 13). Level-Up Your Scenario-Based Learning With VR. Talespin. Retrieved from https://www.talespin.com/blog-post/level-up-your-scenario-based-learning-with-vr
57. Torrence, K. (2023a). 5 Models for Data-Driven Learning. Association for Talent Development. Retrieved from https://www.td.org/td-at-work-guide/5-models-for-data-driven-learning
58. Torrence, K. (2023b, May 21). Tips and Templates: Learning Engineering for Virtual Reality (Conference Presentation). ATD International Conference & Expo, Online.
59. Torrence, K. (2023c, October 24). Measuring Success in Virtual Reality. ARVR for Learning Summit, Las Vegas, NV, USA.
60. Torrence, K. (2023d, October 24). Storyboarding in Virtual Reality. ARVR for Learning Summit, Las Vegas, NV, USA.
61. Torrence, K., & Mueller, R. (2022). Industry Experts Share Learning Experience Design Advice. The Learning Guild.
62. Villanueva, K., & Vaidya, J. (2019). Power up Learning With Mobile Games: The Educational Arcade–An Innovative Approach to Learning. In D. Mentor (Ed.), Advancing Mobile Learning in Contemporary Educational Spaces. IGI Global. https://doi.org/10.4018/978-1-5225-9351-5
63. Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes. Harvard University Press.
64. Wagner, E., Barr, A., Blake-Plock, S., & Robson, R. (2018 September 10). 7 Things You Should Know About Learning Engineering. Educause. Retrieved from https://library.educause.edu/resources/2018/9/7-things-you-should-know-about-learning-engineering
65. Wang, A., Thompson, M., Uz-Bilgin, C., & Klopfer, E. (2021). Authenticity, interactivity, and collaboration in virtual reality games: Best practices and lessons learned. Frontiers in Virtual Reality, 2, 734083. https://doi.org/10.3389/frvir.2021.734083
66. Webster, J., Trevino, L. K., & Ryan, L. (1993). The dimensionality and correlates of flow in human-computer interactions. Computers in Human Behavior, 9(4), 411–426.
67. Yet Analytics, Inc. (2023). A Simple Way to Publish xAPI Statements from Unity. Unity xAPI Publisher. Retrieved from https://www.yetanalytics.com/articles/a-simple-way-to-publish-xapi-statements-from-unity

Hubbard, D. W. (2014). How to measure anything: Finding the value of intangibles in business. John Wiley & Sons.