Exploring the Application of an Instructional Approach Usability Scale (IAUS) For Improving Integrated STEM Instructional Design

Introduction

As K-12 teachers transition from novice to expert during their teaching careers, they will interact with a variety of instructional approaches. Many K-12 science teachers will eventually settle on some version of project-/inquiry-based learning methodology within a constructivist paradigm (Becker & Park, 2011). Project-/inquiry-based learning can be broadly defined as providing students with a sequence of goals they explore; identifying variables, making predictions, and collecting and analyzing data obtained from self-conceptualized experimental processes. Periodically, success within these student-centered learning contexts relies on learners engaging with procedural (cookbook style) science labs that demonstrate important experimental practices and/or content. Numerous researchers view the application of these procedural science activities within more exploratory learning activities as a means of scaffolding learning as students transition to more autonomous discovery of topics they are interested in knowing more about (Kelley et al., 2021; McDaniel & Einstein, 2005; Struyf et al., 2019).

Since the advent of the STEM (science, technology, engineering, math) acronym, project-/inquiry-based approaches have become synonymous with integrated STEM (Sanders, 2008) and, more recently, with maker-based education (Bevan, 2017; Bevan et al., 2015). The integration of science, technology, engineering, and math (Integrated STEM) may best be defined as “the study and ethical practice of facilitating learning [of science] and improving [math] performance by . . . using and managing appropriate [engineering processes] and technology resources” (Huang et al., 2019, p. 8). This integration often takes the form of computer-based technology being used by students to construct their own scientific explorations using computer-aided design (CAD) software and entrepreneurial technology such as 3D printing. The resulting 21st Century approach to science education fits in nicely with a variety of learning environments and student demographics (Bevan et al., 2015; Purzer et al., 2015; Roehrig et al., 2021; Struyf et al., 2019) and plays a prominent role in the Next Generation Science Standards (Chen & Terada, 2021).

The increased push for STEM integration in K -12 learning environments has led to an increase in advanced digital technology and principles of engineering design being incorporated into project-/inquiry-based learning. Thus, instructional designers who work with science curriculum are often tasked with creating integrated STEM learning experiences for K-12 classrooms. Furthermore, STEM rich K-12 instructional design (ID) often involves creating instructional material that will appeal to learners in all classroom settings and addressing rural, urban, and inner-city learning needs, cultures, and challenges. 

Alignment of Stakeholder Thinking

Instructional designers, like teachers, have pre-existing biases for types of learning experiences and specific learning theories (Honebein & Reigeluth, 2021). Moreover, instructional designer biases and preconceptions may clash with those of K-12 teachers and other education stakeholders. Therefore, it is important for instructional designers to keep this potential misalignment in mind when collecting information regarding their target audience (i.e., teachers and students) if they are to create robust, rigorous, and valid learning experiences. In fact, misalignment between instructional designer and classroom teacher instructional approach preferences may contribute to lackluster enthusiasm for the designed curriculum and hesitancy on the part of teachers to adopt designed instructional material.

One of the most important assessments within product driven fields is usability (Flowers, 2005). Rubin and Chisnell (2008) state at the outset of their treatise on usability testing that “usability is only an issue when it is lacking or absent” (p. 3). They go on to state that the absence of frustration on the part of the user while using the designed product makes it usable. Understanding the perceived usability of new/novel science curricula and instructional approaches to learning science is thought to provide valuable feedback that informs the ID process. Therefore, defining what usable means within the ID field relies on how the end user is defined. Review of existing research illustrates that ID usability focuses on assessing usability of the learning material on the part of the learner (Shaikh et al., 2023; Kim, 2024). Furthermore, when teacher perceptions of usability are assessed, these focus on the usability of the designed student resource (i.e., software and hardware) (Balanyà Rebollo & De Oliveira, 2024; Nazar et al., 2020) exclusively. Assessing usability of the prescribed instructional approach employed by teachers to facilitate student learning is just as important as assessing usability of the designed student resources. 

Instructional approach usability assessment is in line with the role teachers play in analyzing results of implemented approaches, identifying potential design improvements, and making adjustments to instructional designs to maximize usefulness, efficiency, effectiveness, learnability, satisfaction, and accessibility (Smith & Ragan, 2004) within their classrooms. Unfortunately, research predominantly focused on learner usability implies that usability on the part of teachers is inherent within the curriculum or that teachers always play an integral role in curriculum design and development. Thus, there is a lack of instructional approach usability research from which instructional designers can develop ID heuristics. This glaring omission is especially troubling within high-profile educational initiatives related to STEM education. 

Dewey (1910) indicates that an instructional approach consists of the methods and techniques educators use to engage students with course content. Moreover, instructional approach usability can be assessed by collecting feedback on teacher perceptions regarding how well the prescribed STEM content and methodology is integrated and aligns with their teaching preferences. This begs the question: if K-12 teachers are the best source for assessing the usability of implementation strategies, is evaluation of teacher perceptions of K-12 integrated STEM instructional approach usability a worthwhile starting point for conducting ID research?

Integrated STEM Instruction

Designing integrated STEM instructional resources requires balancing the orientation of subject-matter content with prescribed learning materials in order to induce mental processing and retention (Kelley et al., 2021; Rapanta et al., 2021; Sanders, 2008). Subject-matter content and prescribed learning material must be complimentary to the instructional approach employed rather than disconnected or disjointed from one another (McDaniel & Einstein, 2005). McDaniel and Einstein (2005) suggest that a material appropriate difficulty (MAD) framework is most suited for achieving this goal. The three components of the MAD framework are distinguishing the type of processing embedded in the learning experience, sensitivity to inherent affordances of to-be-learned content, and careful analysis of the overlap between embedded processes and inherent affordances. Once this analysis has taken place, instructional designers can identify a desired level of difficulty and eliminate redundant learning experiences that do not contribute to enhanced retention. Integration of MAD within a backward design ID model encourages the development of an ID implementation strategy that is complementary in both the instructional approach and desired outcomes.

Learning science in authentic situations promotes transfer of scientific knowledge and enhances understanding of science content (Kelley & Knowles, 2016). The pedagogical principles behind integrated STEM learning experiences are said to create opportunities to apply inquiry-based learning in more authentic contexts (Becker & Park, 2011; Bybee, 2010; Han et al., 2023; Nadelson & Seifert, 2017; Stohlmann et al., 2012; Stubbs & Myers, 2016). Kelley and Knowles (2016) suggest that documenting intervention development improves the ID processes, especially in the areas of STEM subject integration, promoting scientific inquiry, collaboration, and incorporating project-/inquiry-based learning. It is believed that determining integrated STEM instructional approach usability is one way of improving ID that fosters cross-subject collaboration, inquiry, and learning.

Designing for Authentic Learning

The STEM Content Inclusion framework was born out of desires to improve student learning, collaboration, and inquiry skills while promoting the integration of STEM subjects into a single classroom where the teacher incorporates one or more of the STEM subjects to their area specialization (Kelley et al., 2021). Furthermore, the learning-for-use (LfU) framework is an approach to science education that “can be used to support the design of content-intensive, inquiry-based science learning activities” (Edelson, 2001, p. 355). Here, the purpose is to give students firsthand experience with asking questions, gathering evidence, and analyzing the collected data in authentic scientific contexts. 

As instructional designers develop classroom content, they are expected to create a demand for knowledge or generate curiosity (motivate), provide learners with opportunities to observe relationships and communicate constructed knowledge (construct), and apply new knowledge to novel situations as they reflect on learning experiences (refine). The fundamental tenant of the LfU framework is the learner must be motivated to learn “based on a recognition of the usefulness of [the] content beyond the learning environment” (Edelson, 2001, p. 373). The fundamental tenant of integrated STEM is that learning “must be born out of existing school structure, …schedules, common curriculum, and …learning standards” (Kelley et al., 2021, p. 34).

However, there are challenges and limitations to integrating all four STEM subjects into a single cohesive learning experience. One such challenge is long-term sustainability of the learning experience being vital to engaging learners with STEM subject integration (Ejiwale, 2013). Ejiwale (2013) identifies ten barriers to integrated STEM instructional design and application, including lack of qualified STEM teachers, poor content preparation and delivery, poor laboratory facilities, lack of student training, and poor assessment methods. Moreover, teacher comfort level must be taken into account when designing instructional content outside teacher subject matter expertise (Stohlmann et al., 2012). Ensuring instructional approaches build on familiar teaching strategies while developing cross-content knowledge eases the burden placed on teachers as they familiarize themselves with integrated STEM facilitation needs (Dare et al., 2018; English, 2016; Holincheck & Galanti, 2022; Sandall et al., 2018).

Designing integrated STEM Instructional

There is an overarching complexity within the ID process, which necessitates concentrated attention to individual micro-learning components of an integrated STEM ID project as well as the totality of micro-learning components as a single learning experience (Schmidt & Huang, 2021). Micro-learning components are the individual student activities a teacher institutes to promote engaging with the broader content behind identified learning objectives. As the ID process unfolds, the designer defines, or at least mentally constructs, an expansive view of the learning environment and then organizes pedagogical best practices necessary to facilitate expected micro-learning outcomes and achievement (Sims, 2006, 2015). Furthermore, as the shared collection of knowledge necessary to be considered literate in a given STEM subject increases, there is added pressure placed on instructional designers to streamline learner access to the expanding content. 

Educational research, theory, and application identifies two competing educational paradigms within the ID field; teacher-centered education and student-centered education. Much is being written about the plaudits of each paradigm in assisting learners as they construct knowledge (Patel-Junankar, 2021). However, at the center of this discussion sits the often-forgotten component of instructional facilitator aptitude. Efforts to distill content to its necessary components often lead to more explanatory didactic approaches to doling out science and math content in carefully measured increments. Students’ success with regurgitating “presented” content validates the didactic instructional approach and contributes to misconceptions of pedagogical competency (Caprara et al., 2006). Thus, if ID efforts within K-12 curriculum development are to move beyond existing student and teacher comfort levels, then instructional approach usability assessment must be examined. It is the authors’ contention that feedback from this type of ID evaluation process will assist instructional designers in addressing misalignments between proposed implementation strategies and teacher desires and preferences. Moreover, pairing instructional approach usability assessment with existing ID evaluation efforts has the potential of improving curriculum credibility and relevance. 

When developing integrated STEM curriculum, there are a litany of design/product decisions instructional designers must make as they determine which engineering processes and technological resources will best match the science content and mathematical analysis of experimental data. These decisions begin with inception of the integrated STEM learning experience and continues on through to educator adoption of the integrated STEM curriculum at project completion. Many of these design decisions can be handled using ID heuristics; the taking of chances backed by past successes and a vast knowledge base. At other times, instructional designers rely on emerging knowledge, existing ID theories, ID model protocols such as backwards design, and trusted mentors (Ertmer et al., 2009). In each instance, an understanding of target audience perceptions of usability is critical.

ID and Usability

Determining usability on the part of the end-user is an inherent part of every design-based career field, two of which are Engineering and ID. It stands to reason that usability assessment would be a vital part of ID projects involving engineering processes. Pre-career training programs and in career professional development within product-fruition disciplines place considerable emphasis on the interplay between design, function, and the product user (Flowers, 2005). Usability assessment is thought to be a good way for designers to develop a more critical eye toward the proposed end-product (Flowers, 2005; Rubin & Chisnell, 2008). This is also true for the ID process which involves integrating a plethora of pedagogical and methodological practices tied to “learning theories, systematic analysis, educational research, and classroom management methods” (Ashton, 2014, p. 53) into a final educational end-product.

Usability has been defined as a measure of the “general quality of the appropriateness of a purpose [for] any product” (Brooke, 1986, p. 189). The industry standard for conducting quick, efficient usability analysis is the Brooke 1986 Systems Usability Scale (SUS) (Lewis, 2018). The SUS is a 10 question, five-point likert scale with alternating positive and negative items to be addressed by responders. The SUS has been shown to consistently identify areas of user difficulty and struggle, proving itself invaluable to those deploying the SUS during the design stages of product development and afterwards (Blattgerste et al., 2022; Brooke, 1986; Lewis, 2018). The SUS has also been used to assess learner usability within educational settings employing mobile devices and gaming applications (Nazar et al., 2020). But what of instructional approach usability analysis? A lack of usability analysis associated with instructional approach development within printed research begs the question: what impact would usability analysis have on the design and organization of integrated STEM curriculum? 

Purpose and Research Questions

The research described in this paper frames the conscious use of an instructional approach usability scale (IAUS) within integrated STEM instructional design (ID). An exploration into the relationship between instructional approach usability and integrated STEM curriculum design is grounded in two juxtaposed assumptions; (1) in-service teacher responses on an instructional approach usability scale (IAUS) will inform the ID process and lead to discovery of core components necessary for integrated STEM content construction, (2) teacher perceptions of usability will contribute to improved adoption of the designed integrated STEM content. K-12 curriculum passes through a vetting process where subject-matter teachers determine best fit between their preferred teaching style and their perceptions of their students’ preferences. Furthermore, understanding how teachers view curriculum from a perspective of instructional approach usability can have a positive impact on the ID process. 

Identification of a quick survey that elicits strong positive and negative responses by survey responders has the potential of providing instructional designers immediate, and productive, feedback when creating curriculum using unfamiliar technology and/or methodologies. Moreover, application of an IAUS will positively contribute to the ID field, and educational research at large, by shedding light on the application of ID model/theory foundational building blocks. It is the authors’ contention that identification of a simple, yet reliable, instructional approach usability assessment will provide instructional designers useful information relevant to learning experiences involving the integration of two or more STEM subjects. Thus, a comparative analysis of Brooke’s SUS strengths and weaknesses relative to instructional approach (rather than system) will shed light on a readily available resource for novice and experienced instructional designers alike.

This mixed methods case study involves collection and analysis of quantitative and qualitative data to explore the following research questions:

1. What role does usability play during in-service teacher evaluation of instructional content integrating STEM?

2. What aspects of usability analysis make it applicable for improving the learning activity instructional approach embedded within the ID project implementation strategy?

Research Context

After a 1-hour professional development session on a newly designed integrated STEM curriculum, participants were given a 10 question Likert survey and responded to semi-structured interview questions regarding their survey responses. During the 1-hour professional development session, participants were familiarized with a novel instructional approach requiring teachers to facilitate mastery of foundational principles of genetics and exploration of natural selection using an imaginary population of “motorized organisms.”

The newly designed integrated STEM curriculum is a semester-long “Wobble Bot” project where teachers are challenged to guide students as they take on the role of designer to create a fictitious organisms known as a Wobble Bot. Student resources describe these mysterious creatures as motorized polylactic acid (PLA) organisms that rely upon alkaline chemistry to perform all life functions. The project begins with teachers guiding students as they use images and physical descriptors of an “adult” wobble bot to create a 3-dimensional (3D) image using Tinkercad modeling software, with the goal of creating the most accurate digital representation of a parental (P1) wobble bot as possible. 

Next, the instructional approach guides teachers as they facilitate students discovering the genetics of their adult wobble bot and the creation of gametes that are used to generate F1 (filial generation 1) offspring. Teachers are asked to then mentor students as they “design” their F1 wobble bot offspring based on inherited genotype and skills gained form designing their P1 wobble bot. These are then 3D printed, motorized, and run through three survival competitions to determine which organisms survive to “mating.” Students use F1 mating pairs to determine F2 (filial generation 2) genetics for designing their F2 wobble bots based on inherited genotype and lessons learned from first round of natural selection competitions. Finally, armed with this information, the instructional approach tasks teachers with challenging students to create a “most fit” wobble bot preparatory to a second round of natural selection competitions and determination of F3 (filial generation 3) offspring genotype and phenotype.

Throughout the instructional approach, teachers guide student learning of essential biology topics related to genetics and natural selection (e.g., meiosis, patterns of inheritance, Hardy-Weinberg assumptions, mutation, protein synthesis, etc.), repeatedly assisting students as they interact with Tinkercad modeling software and 3D printing (P1, F1, F2, and F3 organisms), explore basic circuitry (i.e., wiring of DC motor), and conduct mathematical analysis of wobble bot population allele frequency and inheritance patterns at the classroom level (15-24 individuals). 

Methods

The first step in analyzing teacher perceptions of usability when they evaluate a STEM instructional approach is to identify an applicable resource for measuring usability. The Brooke’s SUS close-ended usability scale was chosen because of its universal acceptance across multiple design fields (Lewis, 2018). Understanding teacher thought process while completing the usability assessment necessitates asking open-ended questions and allowing teachers to elaborate. The combination of closed-ended and open-ended data collection is characteristic of a mixed methods research approach (Creswell & Creswell, 2017). 

Next, a backwards designed integrated STEM learning experience was chosen because backwards design ID is grounded in identifying desired results and acceptable evidence followed by planning experiences and instruction to achieve stated outcomes (Wiggins & McTighe, 2005). This ID model strategy is in keeping with the premise that usability feedback will inform the design of how curriculum elements are assembled into an implementation strategy and associated instructional approach. Finally, a learning experience involving the integration of all four STEM subjects was chosen because the prescribed complex implementation strategies associated with integrated STEM curriculum make it an ideal backdrop for evaluating instructional approach usability assessment tools.

The collection of both qualitative and quantitative data allowed researchers to more deeply explore K-12 in-service teacher perceptions of usability by analyzing and interpreting participant responses to IAUS questions in light of their elaboration during a 30-minute semi-structured interview. During a 1-hour training session participants were provided prototype designs of developed instructional material and asked to evaluate each student-focused component of the ID project and the instructional approach on a continuum from strongly disagree (1) to strongly agree (5). Emphasis during training session was placed on familiarizing participants with designed content followed by asking them to provide a usability analysis tied to both the content (genetics) and the process (designing vibrating robots). Furthermore, participants were told that IAUS data and interview feedback would be used to inform ID project stakeholder thinking and improve end-product usability. 

Participants

The study involved 11 in-service grade 8 through grade 12 science teachers (three male and eight female) from across the state of Indiana (US) who reported prior experience with STEM lesson facilitation, the STEM activity implementation strategy adopted in their classroom, and willingness to participate in a semi-structured interview. In addition to actively looking for STEM curriculum, each participant voiced an interest in adopting more STEM education and integrating STEM learning experiences within their courses. Two individuals became the focus for a case study analysis based on their different approaches to STEM implementation practices in their classrooms.

Abi is a biology teacher in a high school (40% non-white enrollment) within walking distance of a Mid-West R1 University and has been teaching for 20+ years.  She just recently participated in an integrated STEM learning experience with the school’s Principles of Engineering (POE) instructor, where students designed fishing lures based on unique characteristics of assigned aquatic ecosystem flora and fauna. She found the application of genetic inheritance of organism traits versus one imposed upon the model by designer preferences intriguing. Her traditional method of exploring genetics and natural selection involves “You’re stuck on a planet. Here are the things you have to survive, and you have to create . . .” She uses a lot of storytelling “. . . tying in popular culture and asking students to unravel the story using topics from genetics and natural selection.”

Shelby is a middle school (grade 6 to grade 8) science teacher and has a single biology class for students earning credits for high school. Her school is predominantly white, middle-class, rural American students. Her passion is physical science and doing hands-on learning activities with her students. She has been teaching for just over 10 years, all of which have been at the same school. Shelby integrates one engineering activity into her physical science courses every quarter (8 weeks). Most of these involve something to do with Newtonian mechanics, “and so venturing into something like this would provide continuity between [her] 7th grade classes and 8th grade classes,” which includes biology. She generally uses a lot of PowerPoints and textbook assignments when it comes to genetics and natural selection stating, “there just isn’t a lot out there when it comes to genetics unless you spend a ton of money on science kits, which we don’t have the budget to do.”  

Shelby “loves technology” and was instrumental in her school building a makerspace. “I worked with so many people to learn all the things that would be needed to create our makerspace, and now I am basically the de facto head of it.” Abi, on the other hand, is not “well versed in technology outside of computers, smartphones, and tablets . . .” indicating she “. . . relies on the POE teacher whenever [she wants] to add other forms of technology.” 

Data Collection

A set of 10 Likert survey questions, adapted from Brooke’s 1986 SUS, were used as a benchmark for identifying initial usability of the designed integrated STEM learning experience (Appendix A). Brooke’s 1986 SUS items were developed based on the most extreme response options, with an intercorrelation between items of ±0.7 to ±0.9 (Jordan et al., 1996). An added benefit to the items is their ability to evoke a common strong agreement for one half of the items and a strong disagreement for the other half.  The only change made to Brooke’s SUS was replacing the term “system” with “instructional approach.” Thus, the statement “I think I would like to use this system” became “I think I would like to use this instructional approach.” 

What is now referred to as an Instructional Approach Usability Scale (IAUS) was assumed to have a comparable intercorrelation value for the purposes of this study. It was believed that the IAUS would provide researchers timely feedback during the ID process and prior to pilot testing curriculum to a wider audience involving teachers and students. Furthermore, IAUS responses serve as a good indicator of K-12 teacher affinity for the final ID product within the target community of grade 8-12 science classrooms.

The odd numbered items within the IAUS provide instantaneous feedback on how well the instructional approach is liked, easy to use, integrated, easy to learn how to use, and contributes to confidence and by extension competence. The even numbered IAUS items inform perceived complexity, inconsistencies, cumbersomeness, and need for technical support and relevant training necessary for approach implementation. Information from both sets of questions is anticipated to establish a solid benchmark from which modifications to design, development, and implementation of the integrated STEM approach may be measured. 

The IAUS was followed by a semi-structured interview where responders were asked to explain their ranked responses for each of the ten questions. A case study approach was chosen to best communicate results of two individuals based on their polar approaches to integrating STEM learning into their classrooms. Shelby sees integrated STEM as “an ideal way of engaging learners with science content in authentic ways, and that directly relate to their lived experiences” and feels confident in her ability to facilitate integrated STEM as a solo teacher. Abi is more conservative, seeing integrated STEM as “an opportunity to collaborate with POE teachers who are better equipped to teach engineering concepts mandated by school administration.” Both Abi and Shelby indicated having extensive prior experience with including STEM lessons in their courses, Shelby as a solo teacher and Abi in partnership with POE teachers.

IAUS Findings

Each transcript was reviewed for explicit responses to why participants rated each question the way they did followed by coding for implicit connections between classroom practice and IAUS responses. Findings were discussed between researchers and consensus was reached as to explicit and implicit connections. Quantitative analysis of IAUS numerical rankings were organized by the number of individuals indicating their level of agreement with each statement (1-strongly disagree to 5-strondly agree). The IAUS numerical rankings are describe below.

Figure 1

IAUS Question 1 Participant Responses

Seven of the eleven IAUS participants (63.6%) indicated a likelihood of using the instructional approach to learning genetics and natural selection. Both Abi and Shelby indicated strong agreement with wanting to integrate the learning activity into their existing course content. This is in keeping with Shelby and Abi’s preferences for integrated STEM curriculum that aligns with course content and prior STEM integration experience.

Figure 2

IAUS Questions 2 and 3 Participant Responses

When it came to analyzing the complexity and ease of use, on the one hand slightly more disagreed with the instructional approach being unnecessarily complex (45.5% versus 27.3%). However, responses to ease of use were more ambivalent (54.5% neither agreeing nor disagreeing). Abi and Shelby indicated they did not find the integrated STEM materials unnecessarily complex (both selecting 2- disagree) and agreed (4) that the designed integrated STEM approach would be easy to use. It is reasonable to assume Abi and Shelby’s previous experience with STEM integration lends credibility to their judgement of instructional approach/resource complexity and usability.

Figure 3

IAUS Question 4 Participant Responses

The majority of IAUS responses (63.6%) indicated the need for technical assistance with instructional approach. Abi and Shelby were on opposite ends of the spectrum when it came to the need for technical assistance with Abi indicated strong agreement with needing technical assistance and Shelby strongly disagreeing with this need. These responses are in keeping with Abi and Shelby’s classroom practice and preferences for integrating STEM activities into course content.

Figure 4

IAUS Questions 5 and 6 Participant Responses

When it came to how well STEM subjects were integrated, nine of the eleven participants (81.8%) indicated agreement or strong agreement with the way STEM subjects were integrated. Moreover, all participants indicated there were very few inconsistencies with the integrated STEM approach as designed. Abi and Shelby’s responses were aligned with regards to organization and consistency; Abi strongly agreeing (5) with organization while Shelby indicated only agreement (4). On the other hand, Shelby strongly disagreed (1) with finding “too much inconsistency” within the designed approach while Abi merely agreed (2). It is reasonable to assume Shelby was looking more closely at integration and organization as a sole teacher where Abi was looking at it from a team-teaching perspective.

Figure 5

IAUS Question 7 Participant Responses

When it came to deciding how likely their peers were to learn to use the integrated STEM instructional approach, participants were noncommittal with only one more indicating peers would learn to use the approach than those thinking it would be challenging for their peers to learn to use the approach. Abi and Shelby expressed opposite opinions about whether their peers would be able to learn to use the approach, with Shelby agreeing (4) they would while Abi disagreed (2). The disparity between Shelby and Abi’s response to this question is consistent with their approach to integrating STEM activities into their classrooms. Shelby is confident in her ability and can be seen as projecting that confidence onto her peers while Abi relies heavily on collaborating with engineering teachers and can be seen as projecting similar attributes onto her peers.

Figure 6 

IAUS Questions 8 and 9 Participant Responses

None of the participants determined that the integrated STEM approach was cumbersome, however one individual lacked confidence in using the approach within their classroom as designed. Abi and Shelby were aligned when it came to cumbersomeness of the instructional approach (Abi indicating she disagreed and Shelby indicating strong disagreement). Their confidence with implementing the approach in their classrooms (both strongly agreeing) is in keeping with Abi and Shelby’s previous experiences with STEM integration; prior successes contributing to increased confidence in STEM activity facilitation.

Figure 7

IAUS Question 10 Participant Responses

When it came to possessing the knowledge and skill to implement the integrated STEM approach, it was split 50-50 on those who felt they would need additional training versus those who felt they already had the knowledge and skills to facilitate the student learning objectives. Both Abi and Shelby indicated that the instructional approach would not require them to “learn a lot of things” prior to implementing the instructional approach with Shelby feeling more knowledgeable than Abi (1 versus 2 respectively). Abi’s decreased disagreement is in keeping with her collaborative learning while Shelby’s strong disagreement is in keeping with her solo teacher approach to integrating STEM.

Interview Findings

Each interview began with the prompt, “Tell me your thoughts about the wobble bot lesson material.” This was followed up with questions that connected some aspect of each participant's response, tying it to the IAUS survey questions, such as “Explain your response of [XXX] to how easy you thought the instructional approach would be to use in your classroom.” This process continued for a total of 30 minutes. Each participant was then given the opportunity to share “any feedback [they] think will be helpful to the design team to improve the instructional approach.” Enthusiasm for an integrated STEM instructional approach was shared by all participants during the training sessions, with participants having a variety of approaches for including STEM lessons in their classrooms. 

During Shelby’s interview she expressed a more profound interest in adopting the wobble bot learning activities as a solo instructor, confident in her ability to facilitate STEM integration without the need for math, engineering, and technology teacher input “though welcome”. Moreover, she actively looks for additional integrated STEM learning experiences to include within her courses—her school “actively encourages staff members to integrate making activities into their courses.” Shelby was effusive about wanting to get started with the instructional material right away, even though she would not be covering genetics until the following semester. She indicated she was going to begin “tinkering with the design, 3D printing, and logistics” of content facilitation to “familiarize [herself] with the build process and create reference models for students.”

Abi’s interview demonstrated an attitude towards STEM learning activity adoption tied to collaborating with math, engineering, and technology teachers. Abi’s has a more collaborative view in regards to adopting the wobble bot learning activities and implementing the instructional approach, indicating a desire to talk with the school’s POE teacher to develop a strategy for partnering their students. She felt less confident on being able to implement the designed approach on her own and anticipated “relying on the POE teacher for the heavy lifting of modeling and 3D printing.”

The integrated STEM approach in Abi’s classroom is indicative of what Kelley et al. (2021) refer to as a STEM Content Integration practice while Shelby uses the STEM Content Inclusion model. Moreover, Abi and Shelby indicated integration of STEM subjects as the challenging part of the instructional approach. Abi articulated this dilemma as an organizational need to “do a few hands-on things on the science behind whatever it is we're designing and then using Wednesday, Thursday, and Friday to do prototyping and where we troubleshoot. We do the CAD together and the POE students teach my biology students minimalist CAD design structure and then from that they come up with that initial prototype.” She sees integration as a means of collaborating with the POE who is responsible for engineering and technology while she is responsible for the science and math. Shelby sees this integration as a single teacher ensuring appropriate sequencing of learning such as “designing the object, 3D printing, using electronics and microprocessors, and seeing the final product come to life.” 

Discussion

Research question #1 (RQ1) asks, “What role does usability play during in-service teacher evaluation of instruction content integrating STEM.” Both Abi and Shelby indicated that the purposeful design of all four STEM subjects being integrated into the learning activity is what they find the most intriguing. Both indicated that they normally only combine two subjects at the same time: science and math, science and engineering, or science and technology. Thus, teacher understanding of, and preference for, STEM integration is thought to influence their responses to IAUS questions in a way that reflects their classroom practice. Abi is indicative of individuals who see STEM as a collaborative effort between a team of teachers–each one specializing in a STEM subject. From her responses, the answer to RQ1 would be evaluating usability analysis of Integrated STEM learning activities and instructional approach plays a role in determining how best to support each team member by “knowing which bits and pieces of their specialty need to be added into my classroom.” 

Abi and Shelby emphasized that instructional designers need to purposefully implement teaching strategies, design principles, and expected outcomes from each STEM subject being integrated. When viewed through the lens of RQ1, usability analysis of the instructional approach ensures that course specific integrated STEM curriculum helps eliminate, or significantly reduce, barriers associated with teaching strategies outside domain specific expertise. Thus, IAUS feedback becomes a pivotal data point for defining learning outcomes and encouraging instructional designers to apply design strategies aligned with specific STEM pedagogical and methodological principles and practices (Wiggins & McTighe, 2005) that are not cumbersome or overly complex.

Application of an IAUS when creating the integrated STEM learning activity, in conjunction with a formative STEM framework (White, 2023) appears to establish an informed learning experience that is usable as a collaborative team of teachers or as a single classroom instructor. Both Abi and Shelby indicated the final design helps them align course learning goals with the developed STEM activities, creating motivational and inclusive learning environments that integrate assessment into the learning rather than as a culminating activity. They also recognized that real-world problem-solving requires “skills that cut across disciplines” (Shelby) and “implementing an integrated STEM approach restructures the learning experience” (Abi) such that learning requires “proficiency with multiple resources and forms of knowledge” (Abi and Shelby). Utilization of IAUS feedback ensures that instructional designers are cognizant of teacher working knowledge and how to introduce specialized knowledge outside domain specific expertise (Wiggins & McTighe, 2005). Instructional designers are then able to create instructional approaches that allow teachers to focus on their subject content using contexts from other STEM disciplines to make curriculum more relevant (Kelley et al., 2021).

When looking at the second research question (RQ2), “what aspects of usability analysis make it applicable for improving the learning activity instructional approach embedded within the ID project implementation strategy,” both Abi and Shelby indicated that teacher friendly instructional approaches make it easier for them to concentrate on engaging their students in more hands-on learning experiences. This is in line with previous research showing that if learners are not actively involved in some way, educators feel learning is not happening (Ashton, 2014). 

When classroom teachers are comfortable using an instructional approach, they are able to place more focus on learners by “offering explanations to scientific phenomena” (Abi), “describing design thinking” (Shelby), or “express data in various ways” (both Abi and Shelby). Therefore, one response to RQ2 would be that instructional approach usability feedback will assist instructional designers in refining teacher friendly instructional approaches. This would ensure teachers are unincumbered by complex instructional approaches and can support students as they access prior learning from multiple STEM subjects. The struggle students must work though as they transfer knowledge from one subject to another relies on teachers being able to facilitate this transition (Nadelson & Seifert, 2017). Thus, teacher perceptions of usability, aptitude, and competence with each STEM subject must be at the forefront of the ID process when developing an integrated STEM instructional approach.

Finally, there were numerous instructional approach usability aspects specifically touched on by Abi and Shelby. Their comments included simplified language within documents, lesson plan readability, and engaging students in the design process early and repeatedly. The positive comments regarding the designed curriculum were paired with aspects they indicated needed work, such as the math connections. Here, Abi and Shelby suggested finding ways to have students discover the math rather than just telling them how to determine probability of offspring genetics or the frequency of alleles within the population. Another major point both hit on was the organization and complexity of content so that students were supported from design to 3D printing, and on to assembly of their wobble bot. Shelby stated this as.

Helping my students use knowledge from one subject like engineering within a science problem will give them a chance to use their knowledge in authentic ways. It is about problem-solving in real-life.

Abi stated it this way. 

Breaking the material down by sequential activities allows students to learn the connections between subjects. It will allow students to see that different teachers are pretty much the same, doing the same things. When I teach the science and the POE teacher teaches the engineering right after, and then the math teacher comes in . . . Students see it all come together.      

Implications for Instructional Designers

Science, technology, engineering, and math disciplines are distinguishable by both content and pedagogy. It therefore becomes incumbent upon the instructional designer to bridge gaps between these subjects in relevant and meaningful ways. Engaging science students with math, engineering, and technology requires science teachers to incorporate knowledge from each subject cohesively. It is not the number of subjects being integrated but rather the connections being made to the relevant topics, formulas, principles, etc. that matter (Roehrig et al., 2021). Instructional design approaches that assist students (and teachers) make sense of their lived experience by engaging in activities that provide insight as to how science, technology, engineering, and math are related and support one another is key (Flood et al., 2020).

Instructional approach usability assessment is one way to verify proposed bridges and connections students are expected to make between STEM subjects will ensure learning experiences are not overly complex and cumbersome for teachers. When teachers have a favorable perception of the instructional approach, they will be more likely to promote adoption of learning activities into existing course content. Both Abi and Shelby indicated that teachers are not looking for more things to do, they are looking for more efficient ways of doing what they are already doing. By combining exploration of scientific laws and natural phenomena with technology and engineering’s workable solution problem-solving and mathematical analysis teachers can address real-world scenarios intentionally and in ways that bridge learning gaps between STEM subjects.

Purposeful planning of integrated STEM instructional approaches and ensuring usability on the part of classroom teachers will reduce teacher dependency on learning specialty content outside their subject matter expertise. Usable integrated STEM instructional approaches will reduce the time teachers spend familiarizing themselves with isolated teaching practices, content, and one-off pedagogical/methodological learning practices and afford them more time devoted to supporting mastery of course content. Even teachers with less enthusiastic perceptions of integrated STEM learning will recognize the value of instructional approach usability testing and perhaps even become inclined to integrate the learning activity into their classroom as they reflect on the implementation strategy more deeply.  

Limitations

This manuscript is focused on what information instructional approach usability assessment can provide instructional designers and the ID process. The authors note that using a modified version of Brooke’s SUS does not guarantee the same intercorrelation of items. However, for the purposes of determining if such a measuring tool is feasible for evaluating instructional approach quickly, it has performed as expected. This is an area of further research that can be done, in addition to determine the exact types/wording of survey items to determine instructional approach usability. Moreover, this study focuses on two individual’s responses to IAUS items rather than a larger population. The decision to spotlight these two individuals was done because of their polar views of incorporating STEM curriculum into their classrooms. By default, the results highlighted are not transferrable across a wider audience. However, they do serve as a starting point for further investigation into the study and application of usability assessment within integrated STEM curriculum development specifically and the broader field of ID in general.  

Disclaimer

This research project was conducted with IRB approval and was not conducted using any research funding source.

References

1. Ashton, J. (2014). Barriers to implementing STEM in K-12 virtual programs. Distance Learning, 11(1), 51–57. https://www.proquest.com/docview/1549546817?pq-origsite=gscholar&fromopenview=true&sourcetype=Scholarly%20Journals
2. Balanyà Rebollo, J., & De Oliveira, J. M. (2024). Teachers’ Evaluation of the Usability of a Self-Assessment Tool for Mobile Learning Integration in the Classroom. Education Sciences, 14(1), 1. https://www.mdpi.com/2227-7102/14/1/1?trk=public_post_main-feed-card_reshare-text
3. Becker, K. H., & Park, K. (2011). Integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students’ learning: A meta-analysis. Journal of STEM Education: Innovations and Research, 12(5). (Huang et al., 2019, p. 8)
4. Bevan, B. (2017). The promise and the promises of making in science education. Studies in Science Education, 53(1), 75–103. https://doi.org/10.1080/03057267.2016.1275380
5. Bevan, B., Gutwill, J. P., Petrich, M., & Wilkinson, K. (2015). Learning through STEM-rich tinkering: Findings from a jointly negotiated research project taken up in practice. Science Education, 99(1), 98–120. https://onlinelibrary.wiley.com/doi/full/10.1002/sce.21151
6. Blattgerste, J., Behrends, J., & Pfeiffer, T. (2022). A web-based analysis toolkit for the system usability scale. Proceedings of the 15th International Conference on Pervasive Technologies Related to Assistive Environments, 237–246. https://doi.org/10.1145/3529190.3529216
7. Brooke, J. (1986). System usability scale (SUS): A quick-and-dirty method of system evaluation user information. Reading, UK: Digital Equipment Co Ltd, 43, 1–7. https://www.researchgate.net/publication/228593520_SUS_A_quick_and_dirty_usability_scale
8. Bybee, R. W. (2010). Advancing STEM education: A 2020 vision. Technology and Engineering Teacher, 70(1), 30–35. https://www.proquest.com/docview/853062675/abstract/A4EC0E6B8AA4A00PQ/1
9. Caprara, G. V., Barbaranelli, C., Steca, P., & Malone, P. S. (2006). Teachers’ self-efficacy beliefs as determinants of job satisfaction and students’ academic achievement: A study at the school level. Journal of School Psychology, 44(6), 473–490. https://doi.org/10.1016/j.jsp.2006.09.001
10. Chen, Y. C., & Terada, T. (2021). Development and validation of an observation‐based protocol to measure the eight scientific practices of the next generation science standards in K‐12 science classrooms. Journal of Research in Science Teaching, 58(10), 1489-1526. https://onlinelibrary.wiley.com/doi/full/10.1002/tea.21716
11. Creswell, J. W., & Creswell, J. D. (2017). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications. https://www.google.com/books/edition/Research_Design/335ZDwAAQBAJ?hl=en&gbpv=1&printsec=frontcover
12. Dare, E. A., Ellis, J. A., & Roehrig, G. H. (2018). Understanding science teachers’ implementations of integrated STEM curricular units through a phenomenological multiple case study. International Journal of STEM Education, 5(1), 4. https://doi.org/10.1186/s40594-018-0101-z
13. Dewey, J. (1910). Science as subject-matter and as method. Science, 31(787), 121–127. https://link.springer.com/article/10.1007/BF00487760
14. Edelson, D. C. (2001). Learning-for-use: A framework for the design of technology-supported inquiry activities. Journal of Research in Science Teaching, 38(3), 355–385. https://doi.org/10.1002/1098-2736(200103)38:3<355::AID-TEA1010>3.0.CO;2-M
15. Ejiwale, J. A. (2013). Barriers to successful implementation of STEM education. Journal of Education and Learning (EduLearn), 7(2), Article 2. https://doi.org/10.11591/edulearn.v7i2.220
16. English, L. D. (2016). STEM education K-12: Perspectives on integration. International Journal of STEM Education, 3(1), 3. https://doi.org/10.1186/s40594-016-0036-1
17. Ertmer, P. A., York, C. S., & Gedik, N. (2009). Learning from the pros: How experienced designers translate instructional design models into practice. Educational Technology, 49(1), 19–27. https://www.jstor.org/stable/44429640
18. Flood, V. J., Shvarts, A., & Abrahamson, D. (2020). Teaching with embodied learning technologies for mathematics: Responsive teaching for embodied learning. ZDM, 52(7), 1307-1331. https://doi.org/10.1007/s11858-020-01165-7
19. Flowers, J. (2005). Usability testing in technology education. The Technology Teacher, 64(8), 17–20. https://www.proquest.com/docview/2196440585?pq-origsite=primo&accountid=13360&sourcetype=Scholarly%20Journals
20. Han, J., Kelley, T., & Knowles, J. G. (2023). Building a sustainable model of integrated stem education: Investigating secondary school STEM classes after an integrated STEM project. International Journal of Technology and Design Education, 33(4), 1499–1523. https://doi.org/10.1007/s10798-022-09777-8
21. Holincheck, N., & Galanti, T. (2022). Are you a STEM teacher?: Exploring K-12 teachers’ conceptions of STEM education. Journal of STEM Education: Innovations and Research, 23(2). https://jstem.org/jstem/index.php/JSTEM/article/view/2551
22. Honebein, P. C., & Reigeluth, C. M. (2021). Making good design judgments via the instructional theory framework. Design for Learning, 231–245. https://edtechbooks.org/id/making_good_design
23. Huang, R., Spector, J. M., & Yang, J. (2019). Introduction to educational technology. In R. Huang, J. M. Spector, & J. Yang (Eds.), Educational Technology: A Primer for the 21st Century (pp. 3–31). Springer. https://doi.org/10.1007/978-981-13-6643-7_1
24. Jordan, P. W., Thomas, B., McClelland, I. L., & Weerdmeester, B. (1996). Usability Evaluation In Industry. CRC Press. https://www.routledge.com/Usability-Evaluation-In-Industry/Jordan-Thomas-McClelland-Weerdmeester/p/book/9780748404605?srsltid=AfmBOorCsWu4t_QnwUmmd7iBauHweNDgUl6xpoRe-WtXggXKBYqsHzHN
25. Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(1), 11. https://doi.org/10.1186/s40594-016-0046-z
26. Kelley, T. R., Knowles, J. G., Jung Han, & Trice, A. N. (2021). Integrated STEM models of implementation. Journal of STEM Education: Innovations & Research, 22(1), 34–45. https://web.p.ebscohost.com/ehost/detail/detail?vid=0&sid=6dd996b9-9bc8-4e0e-9a21-d154dd449fbd%40redis&bdata=JkF1dGhUeXBlPXNzbyZzaXRlPWVob3N0LWxpdmU%3d#AN=150980959&db=a9h
27. Kim, H. Y. (2024, September). Development and Usability Assessment of Virtual Reality-and Haptic Technology-Based Educational Content for Perioperative Nursing Education. In Healthcare (Vol. 12, No. 19, p. 1947). MDPI. https://www.mdpi.com/2227-9032/12/19/1947
28. Lewis, J. R. (2018). The system usability scale: Past, present, and future. International Journal of Human–Computer Interaction, 34(7), 577–590. https://doi.org/10.1080/10447318.2018.1455307
29. McDaniel, M. A., & Einstein, G. O. (2005). Material appropriate difficulty: A framework for determining when difficulty is desirable for improving learning. In Experimental cognitive psychology and its applications (pp. 73–85). American Psychological Association. https://doi.org/10.1037/10895-006
30. Nadelson, L. S., & Seifert, A. L. (2017). Integrated STEM defined: Contexts, challenges, and the future. The Journal of Educational Research, 110(3), 221–223. https://doi.org/10.1080/00220671.2017.1289775
31. Nazar, M., Rusman, R., Putri, I., & Puspita, K. (2020). Developing an Android-Based Game for Chemistry Learners and Its Usability Assessment (pp. 111–124). International Association of Online Engineering. https://www.learntechlib.org/p/217800/
32. Patel-Junankar, D. (2021). Learner-Centered Pedagogy: Teaching and Learning in the 21st Century. Springer Publishing Company. https://connect.springerpub.com/content/book/978-0-8261-7718-6/part/part01/chapter/ch01
33. Purzer, Ş., Goldstein, M. H., Adams, R. S., Xie, C., & Nourian, S. (2015). An exploratory study of informed engineering design behaviors associated with scientific explanations. International Journal of STEM Education, 2(1), 9. https://doi.org/10.1186/s40594-015-0019-7
34. Rapanta, C., Botturi, L., Goodyear, P., Guàrdia, L., & Koole, M. (2021). Balancing technology, pedagogy and the new normal: Post-pandemic challenges for higher education. Postdigital Science and Education, 3(3), 715–742. https://doi.org/10.1007/s42438-021-00249-1
35. Roehrig, G. H., Dare, E. A., Ring-Whalen, E., & Wieselmann, J. R. (2021). Understanding coherence and integration in integrated STEM curriculum. International Journal of STEM Education, 8(1), 2. https://doi.org/10.1186/s40594-020-00259-8
36. Rubin, J., & Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons. https://ebookcentral.proquest.com/lib/purdue/detail.action?pq-origsite=primo&docID=343716
37. Sandall, B., Sandall, D., & Walton, A. (2018). Educators’ perceptions of integrated STEM: A phenomenological study. Journal of STEM Teacher Education, 53(1). https://doi.org/doi.org/10.30707/JSTE53.1Sandall
38. Sanders, M. E. (2008). STEM, STEM education, STEMmania. The Technology Teacher, 68, 20–26. https://www.proquest.com/docview/2190110720?pq-origsite=primo&accountid=13360&sourcetype=Scholarly%20Journals
39. Schmidt, M., & Huang, R. (2021). Defining learning experience design: Voices from the field of learning design & technology. TechTrends. https://doi.org/10.1007/s11528-021-00656-y
40. Shaikh, S., Yayilgan, S. Y., Klimova, B., & Pikhart, M. (2023). Assessing the usability of ChatGPT for formal english language learning. European Journal of Investigation in Health, Psychology and Education, 13(9), 1937-1960. https://www.mdpi.com/2254-9625/13/9/140
41. Sims, R. (2006). Beyond instructional design: Making learning design a reality. Journal of Learning Design, 1(2), 1–9. https://eric.ed.gov/?id=EJ1066491
42. Sims, R. (2015). Revisiting “Beyond Instructional Design.” Journal of Learning Design, 8(3), 29–41. https://eric.ed.gov/?id=EJ1083712
43. Smith, P. L., & Ragan, T. J. (2004). Instructional Design. John Wiley & Sons. https://bcs.wiley.com/he-bcs/Books?action=resource&itemId=0471393533&bcsId=2113&resourceId=4275
44. Stohlmann, M., Moore, T., & Roehrig, G. (2012). Considerations for teaching integrated STEM education. Journal of Pre-College Engineering Education Research (J-PEER), 2(1). https://doi.org/10.5703/1288284314653
45. Struyf, A., De Loof, H., Boeve-de Pauw, J., & Van Petegem, P. (2019). Students’ engagement in different STEM learning environments: Integrated STEM education as promising practice? International Journal of Science Education, 41(10), 1387–1407. https://doi.org/10.1080/09500693.2019.1607983
46. Stubbs, E. A., & Myers, B. E. (2016). Part of what we do: Teacher perceptions of STEM integration. Journal of Agricultural Education, 57(3), 87–100. https://eric.ed.gov/?id=EJ1123039
47. White, S. K. (2023). Making a Framework for Formative Inquiry Within Integrated STEM Learning Environments. In B. Hokanson, M. Schmidt, M. E. Exter, A. A. Tawfik, & Y. Earnshaw (Eds.), Formative Design in Learning: Design Thinking, Growth Mindset and Community (pp. 167–178). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-41950-8_13
48. Wiggins, G. P., & McTighe, J. (2005). Understanding by Design. ASCD. https://ebookcentral.proquest.com/lib/purdue/detail.action?pq-origsite=primo&docID=280441

Appendix A

Instructional Approach Usability Scale (IAUS)

(The System Usability Scale – ©Digital Equipment Corporation, 1986 uses the term “system” instead of “instructional approach” for all items and the term “functions” in #5 instead of “activities”)