The 2022 launch and rapid advancement of generative AI (GenAI), such as ChatGPT, a large language model (LLM), have transformed the technology landscape and shaken educational systems (Teubner et al., 2023). Concerns have risen about how these tools affect learning. Some academic institutions have banned the use of GenAI to uphold traditional academic practices (Alasaidi & Baiz, 2023). Others are concerned that students might misuse GenAI, commit academic fraud, or become overly dependent (Dempere et al., 2023), leading to a decline in academic and cognitive abilities such as critical thinking, analytical thinking, and decision making (Zhai et al., 2024). Leveraging GenAI in ways that advance skills and streamline tasks is imperative as technology advances. Therefore, students need the skills to navigate GenAI responsibly in personal, academic, and professional settings. Despite concerns, it is essential to introduce AI literacy into classrooms, particularly in higher education, given the potential for AI technology to increase in the competitive workforce (Lintner, 2024).
AI literacy includes the AI competencies and skills that everyone, not just those with a computer science background, should develop (Laupichler et al., 2022). Long and Magerko (2020) define “AI literacy as a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace” (p. 2). Research identifies recurring competencies: understanding AI concepts and AI training techniques (Casal-Otero et al., 2023), understanding AI functions, applying concepts and evaluating AI, and ethical considerations (Ng et al., 2021), understanding the architecture of AI, and recognizing limitations (Walter, 2024), addressing privacy, security, and plagiarism, and tracking AI trends (Černý, 2024). Collectively, these frameworks emphasize fundamental knowledge of AI, practical application, and ethical engagement.
While existing research has identified key competencies and frameworks, there remains a need for practical examples demonstrating how to effectively embed AI literacy in higher education. This study aimed to examine the effectiveness of a graduate-level course introducing students to GenAI. In 2023, the professor of a graduate research methods course in International Studies wanted to expose her students to Gen AI. During a week-long asynchronous module, GenAI replaced the teacher. The purpose of this module was for the students to interact with Gen AI and develop a healthy level of skepticism. There were six activities where the students interacted with GenAI and completed discussion boards. The first activity, “Make AI lie to you,” was intended to encourage creativity, explore GenAI through repetitive prompts, and foster a healthy respect for the technology's limitations. The other exercises, “Teach me about a concept,” “Teach me about Standardization,” “Is this a good research question,” “AI & Tableau,” and “Final Thoughts about AI,” were designed to encourage creativity and use GenAI as a sounding board for testing new research ideas and teaching the students complex statistical concepts of which they were unfamiliar.
This study was guided by the following research questions (RQ): (1) How did this module contribute to students’ AI literacy? (2) After completing the module, what was the students’ knowledge, perceived value, perceived cost, intention to use, and trust in GenAI? (3) What content would be beneficial for an AI literacy curriculum for higher education?
This mixed-methods study utilized a post-module survey, discussion board posts, and a focus group. Participants were 12 graduate students (7 females and 5 males; 7 master’s and 5 doctoral). To answer how the module contributed to students’ AI literacy (RQ1), participants' discussion board posts were scored using the Structure of Observed Learning Outcomes (SOLO) Taxonomy, and emerging themes were identified. The SOLO taxonomy provided a framework for evaluating a hierarchy of learning responses that reflect increasing levels of understanding and integration (see Figure 1) (Biggs & Collis, 1982). A post-module survey (Chan & Zhou, 2023) and inductive and deductive coding of the discussion posts were analyzed to address students’ perceptions of the module (RQ2). The survey, based on expectancy theory, consisted of 23 Likert-response items (5-point scale: strongly disagree=1 to strongly agree=5) grouped into four subscales: 1) knowledge of GenAI, 2) perceived value with sub-categories for attainment, intrinsic, and utility value, 3) perceived cost of GenAI, and 4) intention to use GenAI. The authors added four additional questions assessing trust in AI.
Figure 1
SOLO Taxonomy Levels and Characteristics
Across the six AI literacy activities, most participants demonstrated multistructural understanding, indicating the ability to identify and describe multiple yet independent aspects of each activity. Unistructural levels were predominant in the first two activities. In contrast, relational and extended abstract levels, reflecting higher-order synthesis and transfer, appeared less often but increased in later activities. Overall, the data suggest progressive development from basic to more relational and applied understanding of AI concepts (see Figure 2).
Figure 2
SOLO Taxonomy Scores Based on the Discussion Board Posts
Survey results indicate high levels of knowledge of GenAI, and moderate perceived value, with neutral perceptions of perceived cost and intention to use GenAI (See Table 1). Trust in GenAI was moderately low. Intention to Use was significantly correlated (p=.04) with Overall Literacy score (r2=.74) and Trust (r2=.67) (See Table 2).
Table 1
Descriptive Statistics from Post-module Survey
Construct | Mean (1-5) | SD | Lower | Higher |
Knowledge | 4.11 | 0.88 | 3.23 | 4.98 |
Perceived Value | 3.55 | 0.78 | 2.77 | 4.33 |
Attainment Value | 3.73 | 0.79 | 2.94 | 4.52 |
Intrinsic Value | 3.27 | 0.94 | 2.33 | 4.22 |
Utility Value | 3.59 | 0.62 | 2.97 | 4.21 |
Perceived Cost | 3.14 | 1.09 | 2.05 | 4.23 |
Intention To Use | 3.14 | 0.83 | 2.30 | 3.97 |
Trust | 2.73 | 0.92 | 1.80 | 3.65 |
Table 2
Correlations of Post-module Survey
Knowledge | Value | Cost | Intention to Use | Trust | Overall Literacy | |
Knowledge | 1.00 | |||||
Value | 0.01 | 1.00 | ||||
Cost | -0.50 | -0.49 | 1.00 | |||
Intention to Use | 0.15 | 0.79 | -0.24 | 1.00 | ||
Trust | 0.48 | 0.79 | -0.35 | 0.86 | 1.00 | |
Overall Literacy | 0.04 | 0.64 | -0.28 | 0.82 | 0.61 | 1.00 |
Themes from the discussion post analysis are presented (see Figure 3). The emerging theme that sums up participants' perception of GenAI is content vs mechanics. Participants were skeptical about using GenAI to produce academic content. They shared concerns about misuse and academic fraud, which align with those reported by Dempere et al. (2023). The participants expressed fear that using GenAI for content would dampen their voices. As graduate students, they value having their voices present in their work. In addition, participants highlighted that the overuse of GenAI could limit users' ability to consciously learn if they habitually rely on the tool. In contrast, participants found value and intention to use GenAI for mechanics such as spelling and grammar check, brainstorming, and receiving nonjudgmental feedback. One participant stated that GenAI’s outputs inspired them to think more critically. This observation challenges the notion that GenAI reduces critical thinking (Zhai et al., 2024) and raises concerns about overuse. Therefore, crucial aspects of GenAI use are developing healthy habits, which arise from knowledge of GenAI features and limitations, along with the value it can bring to the task at hand. This amplifies the need for AI literacy in higher education.
Another concern hindering GenAI use is distrust. Some participants made blanket statements that they cannot trust anything on the Internet, while another participant stated that “AI might not have all of the information we need...[and] poses a serious problem in respect to what we want AI to perform for us.” However, when asked what could be implemented to increase trust, participants wanted reliable sources for GenAI outputs, institutional policies for GenAI use, and AI literacy. One participant stated, “As technology progresses, if you know how to use it, you’re getting a head of the curve, and if you’re more efficient, you will have an advantage.”
Figure 3
Themes from Discussion Boards
Thematic analysis of discussion board posts and the focus group session identified eight core AI literacy competencies: foundational knowledge of how AI works; awareness of capabilities and limitations; effective prompting; basic technical understanding; evaluating AI output; ethics and social implications; transparency and trust; and self-awareness of AI dependence (see Figure 4).
These findings echo existing AI literacy frameworks, such as Ng et al. (2021) understanding AI functions, applying, and evaluating AI concepts, and Walter's (2024) suggestions to address architecture, limitations, and ethics. Participants in this study reaffirmed these competencies, along with introducing effective prompting and self-awareness of dependence as competencies to include in a higher education AI literacy module.
Figure 4
AI Literacy Components Derived from the Thematic Analysis of this Study
This study reinforces the need to integrate AI literacy into educational settings to equip students with essential GenAI skills. While participants acknowledge the benefits of GenAI, such as brainstorming, feedback, and mechanics, their distrust, concern about over-reliance, and use of GenAI for content hinder their intention to use it. Future studies will incorporate additional learning strategies to bridge this gap to enhance students’ AI literacy.
EdTech Archives