The rapid advancement of generative artificial intelligence (GAI) technologies has prompted significant interest and concern within higher education, calling for immediate attention to AI literacy concerns (Suriano et al., 2025). As higher education incorporates these tools into teaching, learning, and research activities, a need arises to evaluate their impact on student learning, academic integrity, professional ethics, and critical thinking (Kasneci et al., 2023).
Research gaps remain concerning the long-term effects of GAI on students’ critical thinking, creativity, and learning achievement. Currently, there is little exploration of the pedagogical strategies that effectively integrate generative AI into information literacy learning strategies (James & Filgo, 2023). This dearth limits educators' ability to harness these tools for enhancing teaching and learning, specifically for teaching information literacy and research practices (Bouguettaya, 2025; Gu & Yan, 2025).
As information professionals, academic librarians are in a unique position to serve as guides for students using GAI to conduct research. When designing these pedagogical strategies, the behavioral, cognitive, and affective domains of learning should be considered. This proposed study is an effort to begin the examination of the impact of information literacy instruction, coupled with AI literacy instruction, on students' self-perceptions of AI competency.
By exploring pedagogical strategies that enhance learning while mitigating risks, librarians can foster an academic environment that upholds integrity, sustains critical thinking, and fosters creative skills. The future of higher education relies on our ability to assess and adapt to the evolving role of GAI, ensuring that it serves as a tool for empowerment rather than a hindrance to intellectual growth.
This repeated measures test study took place in a face-to-face, undergraduate upper division course at a mid-size university in the Southeastern U.S. Upon IRB approval, students were invited to test the efficacy of an information literacy model, 3P-AIQ, for increasing GAI literacy competency when using GAI to complete a research assignment. Foundational to the study is the proposed three-phase recursive model, 3P-AIQ. Phase 1: Control the Corpus demonstrates the process of selecting appropriate databases, identifying relevant resources, and organizing these resources. Phase 2: The Mind Meld involves the librarian-guided, scaffolded engagement in critical thinking, evaluation worksheet for students to interpret a scholarly article. Phase 3: Talk to Your Robot involves student guidance through the use of a framework to select the right GAI tool for their desired research outcome and the design of strategic prompts to elicit quality GAI output.
The course focused on basic elements of research and evaluation within the health, kinesiology, and sport fields. The primary course objective was to emphasize scope, meaning, and basic concepts of scientific research. Students complete a capstone literature synthesis requiring research question development, finding relevant literature to support the question, evaluating the relevance of the evidence-based literature, and synthesizing the results.
Students within this course only grasp a basic understanding of evidence-based practice and research concepts; as a result, identifying and interpreting relevant literature can be challenging. The accessibility of GAI creates temptation for students to reduce assignment preparation. Irresponsible and unethical use of GAI has become an implementation barrier in the classroom, with the need for guidance in the appropriate use of artificial intelligence being a necessity.
Students attended a session that demonstrated how to use library databases to complete a literature review.
Students conducted a search and selected a relevant article. Using the provided critical review worksheet in class, students conducted a review of the article, considering relevance, currency, research methodology, etc. After instruction, students were shown an explanatory graphic depicting the workflow for using 3P-AIQ to conduct higher education research. No GAI was used during Day 1.
Students took a pretest, the Autonomy and Competence in Technology Adoption Questionnaire (ACTA). ACTA measures self-perception of competency and adoption attitude. This instrument has demonstrated reliable and valid assessment of technology competence (Peters et al., 2018) and was used to measure behavioral intent and user confidence associated with the intervention.
Students from Day 1 attended a session focused on the 3P-AIQ. This instruction provided modeled and scaffolded guidance on the appropriate and ethical use of GAI tools for conducting research.
Students registered for an Elicit.com account and used Elicit AI to generate a review of the article selected on Day 1. Students compared the critical review they created on Day 1 to a critical review of the article generated by Elicit AI. This comparison activity was followed by a discussion highlighting the similarities and differences between the two reviews. This activity modeled a critical and ethical engagement with GAI tools in the research process.
After the comparison activity and class discussion, students completed the ACTA (Posttest).
Descriptive statistics for both pre- and post-item responses were calculated for all items included in the ACTA (see Table 1, supplemental material). To determine the effect of the model-based intervention, Mann-Whitney U-tests were selected to compare student responses from pre- to post-administration. As the participant responses were not paired by respondent, the data collected were ordinal, and the study included a very small sample size, a non-parametric test was selected. The instructor of record collected observation data in the classroom, unguided and unstructured by the researchers.
Table 1 displays the mean scores for each item’s responses from pre to post intervention and test decreased in items 1, 2, 4, 6, 7, 8, 9, 10, and 12, and increased in items 3, 6, 11, 13, and 14. Mean score changes indicate participants believe that the use of GAI tools in the research process can improve their lives. While participant results indicate that they feel pressured to use AI tools in the research process (item 11; M=2.11, SD=1.76), responses indicate that students experienced an increase in confidence using AI tools effectively in the research process (item 13; M=4.11, SD= .93) and an increased level of familiarity and ease with the use of AI tools in the research process (item 14; M=4.00, SD=.87).
Hypothesis testing was conducted using a series of Mann-Whitney U-tests with two-tailed significance comparing pre- and post-test median scores per item. Results are summarized in Table 2. The results indicated that there are no statistically significant differences between pre and post-intervention scores on any item (all p-values > .05). While none of the changes were statistically significant, trends in the data were observed in the pre- and post scores.
While it cannot be said that the intervention was responsible for any statistically significant participant change from pre to post intervention, changes to the item medians and observations of student performance in the classroom revealed findings that warrant examination. Several item responses indicated that participants felt an increased sense of confidence (item 13; Mdn = 11.67, U = 60.00, Z = .841, p = .456) and ease (14; Mdn = 11.67, U = 60.00, Z = .843, p = .456) in using AI tools, indicating potential increases in self-efficacy. Other item responses indicated affective changes in participants; median changes suggested that post intervention, students felt more pressure to use AI tools (item 1; Mdn = 9.94, U = 45.500, Z = -.398, p = .710, item 11; Mdn = 10.61, U = 50.50, Z = .086, p = .941, item 12; Mdn = 9.06, U = 36.50, Z = -1.04, p = .331) and that they would feel “bad” about themselves if they did not (item 6: Mdn = 10.33, U = 48.00, Z = -.120, p = .941). Overall, the intervention appears to have potentially improved competency in some areas, while possibly resulting in a slight negative effect in others.
The quantitative findings are supported by the course instructor's observations. These observations noted that students demonstrated competency in identifying appropriate databases, articles, and the GAI tool application, as evidenced by multiple student assignments, which included articles that were found using the GAI tool. The instructor required students to document within the reference list of their capstone assignment which GAI tool was used, with all but one student incorporating the use of a GAI. Students were noted to discuss literature more confidently, in that the GAI tools used provided support in the form of summaries and explanations of often daunting research findings for a beginning researcher. The first two parts of the capstone literature synthesis, which involved the selection of a research topic and a literature review table, were also completed before the due date, with all but one student aligning with previous research that indicated that GAI can streamline processes for students by serving as a supportive intervention, which in turn, fosters increased motivation for learning (Mallillin, 2024).
This pilot study was conducted to test a proposed intervention. The data collection and research methodology were based on considerations of the small sample size. The sample size in this study could potentially mask any statistically significant participant change. Additionally, participant responses were not paired in the pre and post-tests, and results were calculated by group versus single participant. Finally, the scale selected as the research instrument was not specific to GAI use and adoption. Since the administration of this study, more relevant, validated studies have been published (see Arslankara & Usta, 2024; Stevens & Stetson, 2023; Yilmaz et al., 2024).
While the results of this pilot study are not statistically significant, trends in the data do provide preliminary information that warrants further study. Results suggest that the intervention's effect may not be uniform across all competencies and that further investigation using paired participant samples, a more nuanced, educational-context appropriate scale, and a larger sample size would be beneficial to draw more robust conclusions.
This study begins the investigation into the impacts of GAI on student information literacy competencies. Developing comprehensive pedagogical frameworks for the use of GAI in higher education is essential to address concerns related to security, privacy, bias, and responsible usage. Exploring effective pedagogical strategies for integrating GAI into teaching practice will help educators maximize the benefits of these tools while minimizing potential risks.
This work was created with the assistance of GAI to augment the authors’ original ideas and expertise. Following 3P-AIQ principles, the authors provided the initial concepts, sources, and prompts. All AI-generated content was subsequently reviewed, fact-checked, and edited by the authors, who retain full responsibility for the final text's accuracy and veracity.
EdTech Archives