The main objective of this study was to create a proof of concept for the use of Generative AI (GenAI) tools as co-intelligence Subject Matter Experts (SMEs) in the task analysis phase in the instructional design process. Improving the task analysis component efficiency by reducing the interaction between the instructional designer and the human SME increases the quality of the instructional process and the benefits gained from the task analysis component.
According to Jonassen et al. (1999), task analysis (TA) is one of the most important components in the instructional design process because it helps clarify the outcomes of instruction, helps decide which outcomes should be further analyzed and developed and can help with the arrangement of the instructional components and requirements into the instructional sequence. However, because of the high demand in time and resources, TA is often skipped as part of the instructional design process. TA is even more important in the healthcare field, where busy professionals such as MDs, pharmacists, and nurses have limited time allocated to serve as SMEs. Therefore, we propose that it will be beneficial to integrate GenAI tools as co-intelligence SME (Mollick, 2024; Mollick & Mollick, 2023) in the TA process to potentially reduce the interaction time between instructional designers and SMEs.
To explore the potential of GenAI to be used as a virtual SME, an instructional designer collaborated with a pharmacy faculty that is also a practicing pharmacist. To start, the pharmacist faculty member selected from the O-Net OnLine occupation database (https://www.onetonline.org) the following four pharmacist tasks that are typically replicated during experiential and laboratory pharmacy education in various contextual settings:
For each task selected, ChatGPT-4 was used following a dialogical prompting structure with an initial prompt that provided the context and role parameters for the procedural and cognitive TA, followed by ad-hoc prompts. Below is presented the generic dialogical prompting structure that was adapted for each of the four tasks.
You are a seasoned pharmacist with significant experience in both ambulatory care, community and institutional pharmacy workplaces. You are working with me, an Instructional Designer, taking the role of a Subject Matter Expert that is helping me understand various pharmacist tasks that I will later use to complete a task analysis for the development of instructional activities for pharmacy students and residents. In this context I will like to describe the steps for the task described below.
Task: [insert task description here]
Please focus on a synthetic description of the steps that are part of this task in the context of [insert contextual setting here]. This information will help me complete what is known as procedural task analysis. As a next step, please focus on the complex steps you identified and for each complex step add information that answers why the step is important for the overall task and how that step complements the steps that are before it and after it in the structure of the task. This information will help me complete what is known as cognitive task analysis.
…
Sample ad-hoc prompt:
Please summarize the above task analysis by: (a) providing a synthetic description of each step in the task followed by (b) a synthetic format of the importance for each step.
Below is a sample segment from the output generated by ChatGPT as a synthetic description of the procedural TA (main steps) and associated cognitive TA (step importance), as a result of the prompting for Task 2: Review prescriptions to assure accuracy, to ascertain the needed ingredients, and to evaluate their suitability [ community settings].
1.Verify Prescription Legitimacy & Completeness
- Ensure all required elements of the prescription are present, such as patient and prescriber details, drug name, dosage, quantity, and proper authorization. This step involves confirming that the prescription is legally valid and complete.
- Why is important: This step ensures that the prescription is legally valid and contains all necessary information. Missing or incomplete data can lead to errors or delays in treatment, which can impact patient safety and therapy.
2. Check for Accuracy & Appropriateness
- Evaluate the prescribed medication for correct dosing, frequency, and formulation, considering patient-specific factors like age, weight, and renal function. Ensure the therapy is appropriate based on the patient's medical history and current condition.
- Why is important: It ensures that the right medication, dose, and form are prescribed for the patient. This step prevents potential overdoses, underdoses, or ineffective therapies, ensuring that the prescribed treatment aligns with the patient’s medical needs.
As an overall observation, while the content of the synthetic description of steps followed a similar pattern across the four tasks, the actual format of the output differed between prompt running when the same GenAI tool, ChatGPT-4, was used by the same researcher and the prompt running for each task was executed in a new chat window.
Once the procedural and cognitive TA was completed using ChatGPT as a virtual SME partner, the output quality was analyzed in two steps. First, since pharmacists are busy professionals, a holistic rubric was developed to streamline the evaluation process and increase the effectiveness of pharmacist evaluation of the task analysis quality. Second, to evaluate the potential of ChatGPT to act as a co-intelligence agent for the instructional designer, the synthetic task descriptions, along with the rubric, were then used by pharmacists who had both academic and practice appointments.
The output for each of the four tasks was selected and evaluated once based on the pharmacist’s expertise with the task setting (ambulatory care, community pharmacy, or institutional pharmacy) and covered both procedural and cognitive task analyses. While completing the rubric, pharmacists also provided short notes about the inconsistencies they found and recorded the time needed for the review.
The instructional designer and the pharmacy faculty member developed a basic holistic rubric to quantify the overall quality of both the procedural and cognitive task analyses output generated by the virtual SME, ChatGPT, in this case (Table 1).
The rubric had an additional column, not shown in Table 1 for space consideration, that was used to collect information about the inaccuracies found by the pharmacist reviewers during their analysis of the task analyses generated by ChatGPT, the virtual SME partner.
Table 1
ChatGPT Task Analysis Output Rubric
1-weak | 2- average | 3-strong | |
Procedural Task Analysis | |||
1. AI-generated step sequence (order of steps are appropriate for practice) | 50% or more steps out of sequence | A few (up to 25%) steps out of sequence | All steps in sequence |
1.a. Sub-steps included with appropriate step | 50% or more sub-steps with wrong main step or missing | Up to 25% sub-steps included with wrong main step or missing | All sub-steps included with appropriate main step |
2. Missing TA steps | 50% or more steps missing | Some (up to 25% of total AI steps) missing | No steps missing |
3. How TA steps are actually performed by a pharmacist (e.g., steps included that are performed by other members of the team or are implicit steps in practice) | 50% or more steps are implicitly part of a different step | Up to 25% are implicitly part of a different step | All steps are explicitly used in practice |
Cognitive Task Analysis | |||
4. AI justification of step importance (review after evaluating steps 1 to 3) | Importance for 50% or more steps poorly justified or more appropriately for another step | Up to 25% of steps poorly justified or would more appropriately for another step | Importance for all steps fully justified |
The rubric scores assigned by the practicing pharmacists for all four tasks are summarized in Figure 1 below. As shown in Figure 1, for Task 1, Task 2 and Task 4 the pharmacists evaluated the quality of the virtual SME task analysis output at average or strong levels.
Figure 1
Summary of Rubric Scores
The notes provided for these tasks supported the average-level scores by indicating, for example, the potential of some steps to be interchangeable with other steps (step sequence), missing one step or sub-step, or steps that are implicitly performed by the pharmacists (e.g., the review of proper medication storage requirements) and are not perceived as a stand-alone step in the task.
Task 3 was the only one that was scored as weak because of a significant number of missing steps and had three of the four remaining scores only at averages. The notes from the pharmacist reviewer pointed toward an issue that was not obvious for the instructional designer: this task combined two different types of compounding, sterile and nonsterile, which are significantly different from a practitioner perspective. Due to the combination of these otherwise independent tasks, the task analysis generated by the virtual SME had missing steps. Several steps generated were formulated in a more generic mode for applicability to both tasks combined in Task 3. The recommendation of the pharmacist was to split this task into sterile and nonsterile compounding and run the task analysis independently for each new task.
The average review time for these tasks using the task summary output and the holistic rubric was around 13 minutes, with review time ranging between 10 and 20 minutes.
The proposed process integrates GenAI tools (e.g., ChatGPT) as a virtual subject matter expert (SME) pharmacist rather than replacing the SMEs from the TA stage in the instructional design process. GenAI tools potentially can serve as a co-intelligence virtual partner for instructional designers as well as for pharmacy faculty who want to include a TA when designing their own courses. The output produced by the virtual SME was effective in producing average to strong quality for both the procedural and cognitive task analysis, as evaluated by human SMEs. In addition, the virtual SME was efficient by significantly reducing the time needed to generate procedural and cognitive TA.
However, because this proof-of-concept pilot study was based on a limited number of tasks with one pharmacist reviewer for each task, future research is needed to (a) increase the effectiveness of the rubric used by the human SME reviewer and (b) further test this process for more complex tasks. More importantly, this process needs to be tested with multiple pharmacist practitioners for additional, more complex tasks.
EdTech Archives