Differences in Ill-Structured Task Performance by Learners’ AI Utilization Skills and Prior Knowledge

Introduction

The integration of generative artificial intelligence into higher education represents a paradigm shift comparable to the advent of the internet, fundamentally transforming how learners access information, construct knowledge, and solve complex problems (Bozkurt et al., 2023). However, the widespread adoption of these tools brings a paradox: while access to high-performance AI is becoming democratized, the ability to leverage it effectively remains uneven. We face a new form of “digital divide”—not of access, but of usage—where the mere availability of tools like ChatGPT does not guarantee equitable learning experiences or successful outcomes. This emerging “second-level digital divide” is characterized not by differential access to technology, but by inequalities in patterns of use, strategic engagement, and learning outcomes (Hargittai, 2002; van Deursen & van Dijk, 2014).

However, much of the existing research on AI in education has disproportionately focused on outcome-level indicators while paying limited attention to how learners actually interact with AI systems during problem-solving processes (Crompton & Burke, 2023). The purpose of this study is to examine how learner-related characteristics shape both the outcomes and processes of AI-assisted problem-solving in higher education, with a focus on two key learner variables—prior knowledge and AI utilization skills. To address these objectives, this study employs a mixed-methods approach to answer the following two research questions:

RQ 1. To what extent do learners’ prior knowledge and AI utilization skills predict performance outcomes on an AI-supported ill-structured instructional design task?

RQ 2. How do learners’ prior knowledge and AI utilization skills shape their interaction patterns and learning processes during an AI-supported ill-structured instructional design task?

Theoretical Background

The instructional design task employed in this study is a quintessential ill-structured problem, characterized by indefinite starting states, ambiguous goals, and the absence of a single, algorithmic solution path (Jonassen, 1997; Simon, 1973). In this high-stakes cognitive environment, Generative AI (e.g., ChatGPT) transcends its role as a traditional information-retrieval tool to serve as a “cognitive partner” or “intelligent tutor” (Mollick & Mollick, 2023). By offloading lower-level cognitive processes—such as initial idea generation, drafting, and organizing information—to the AI, learners can theoretically reallocate their cognitive resources toward higher-order thinking, such as critical evaluation and decision-making.

Prior knowledge remains a fundamental prerequisite for effective learning, even in AI-supported environments (Dochy et al., 1999). Learners with high prior knowledge can serve as critical evaluators of AI and guide the AI to produce more sophisticated outputs. Conversely, learners with limited prior knowledge lack the internal standards necessary to assess the AI’s reliability and are more likely to passively accept AI responses as authoritative answers without critical reflection (Parasuraman & Manzey, 2010). This study also identifies AI utilization skill—often operationalized as prompt engineering or AI literacy—as the critical gatekeeper determining the quality of human-AI interaction (Long & Magerko, 2020). Learners with high AI utilization skills can treat the AI as a dynamic thinking partner, engaging in multi-turn dialogues to deepen their inquiry. In contrast, those lacking these skills may be relegated to simple “question-answer” interactions that yield generic results.

Method

A total of 60 learners were recruited for this study through on-campus announcements at a university in South Korea. The participants’ ages ranged from 21 to 40 years, with a mean age of 27.0 years (SD = 4.3).

The task assigned to participants was to develop a lesson plan reflecting constructivist learning principles. During this process, participants were instructed to use generative AI and internet search engines. The experimental procedure consisted of three phases: Pre-assessment, Task Performing (Ill-structured Problem-Solving Task), and Post-assessment and Interview. In the Pre-assessment phase, Prior Knowledge was measured using a constructed-response item asking participants to “Write down everything you know about constructivist learning principles.” AI Utilization Skills were measured by asking participants to “Propose your own strategy for writing prompts when using generative AI,” scored based on the number of effective elements included, using OpenAI’s (2023) effective prompt engineering strategies as the criteria. In the Task Performing phase, participants freely utilized ChatGPT-4o and online search engines during the 60-minute task while screen recordings, AI interaction logs, and video recordings were collected. Subsequently, a Stimulated Recall Interview (Calderhead, 1981) was conducted to explore the learners’ internal thinking processes and specific performance strategies in depth.

Multiple Regression Analysis was conducted, with AI Utilization Skills and Prior Knowledge entered as Continuous Variables. The subjects of qualitative analysis were Multi-modal Process Data, including screen recordings, video recordings, AI interaction logs, and interview transcripts. These data were analyzed in depth at three levels: frequency analysis of behavioral codes, time-series pattern analysis, and sequential process analysis.

Results

Factors Affecting Task Performance

The results of the regression analysis indicated that the derived model was statistically significant [F(2, 57) = 37.610, p < .001], explaining approximately 56.9% of the total variance in Task Performance scores (R² = .569). Both AI Utilization Skills (β = .486, t = 4.717, p < .001) and Prior Knowledge (β = .372, t = 3.612, p < .001) were identified as predictive variables exerting a significant positive (+) influence on Task Performance (see Table 1). The results indicated that AI Utilization Skills (β = .486) possessed stronger predictive power for Task Performance than Prior Knowledge (β = .372). These findings suggest that while both Prior Knowledge and AI Utilization Skills are essential, the ability to interact effectively with generative AI played a more decisive role in determining the quality of the final output in the specific task context of this study.

Table 1

Results of Multiple Regression Analysis

Note. R2 = .569, F (2, 57) = 37.610, p < .001. Dependent Variable: Task Performance Score.

Patterns Characterizing Learning Processes

Qualitative analysis was conducted to uncover the behavioral mechanisms underlying the performance differences identified in the quantitative phase. The analysis revealed that the High AI Skills group exhibited significantly more active and expansive exploratory behaviors than the lower group (see Table 2).

Table 2

Distribution of Behavioral Codes by AI Utilization Skills

Specifically, this group showed a notably higher proportion of Idea-Based Questions (8.90% vs. 3.22%) and Use External Search (10.60% vs. 5.54%), indicating a broader scope of information. This suggests that high AI capabilities enable learners to use the tool not merely to derive answers, but also as a cognitive partner for the expansion of thought. Conversely, the Low AI Skills group showed a high proportion of Copy-Paste AI-output (10.74% vs. 6.22%), and the frequency of Ask Follow-up (11.06%) was high in this group; this is interpreted as reflecting an inefficient interaction pattern characterized by repetitive corrective questions due to a failure to obtain desired results stemming from a lack of sophisticated questioning strategies. In summary, AI Utilization Skills primarily influenced the Agency & Efficiency of tool use; lower skills were associated with an ‘automation bias’ that relied heavily on the tool’s convenience. On the other hand, Prior Knowledge was involved in the Verification & Elaboration of information. Higher knowledge reinforced behavioral characteristics as a ‘Verifier’ and led to critical use of AI.

Discussion

This study demonstrates that learners’ prior knowledge and AI utilization skills decisively shape both the outcomes and the processes of AI-assisted problem-solving. Despite the use of identical tools, significant variations in interaction patterns and cognitive strategies underscore a usage-based digital divide at the process level. Specifically, while high-competency learners engaged in cross-verification and iterative refinement, those with lower skills relied on linear, passive consumption patterns.

A key contribution is the empirical distinction between productive cognitive offloading and unproductive outsourcing. Competent learners utilized AI as a "Cognitive Partner," reallocating cognitive resources toward higher-order evaluation. Conversely, those lacking competencies exhibited an "Automation Bias," treating AI as a substitute, resulting in superficial engagement. This explains why high-quality outputs do not always reflect deep learning, supporting concerns that outcome-based metrics alone are insufficient to capture the quality of learning. Consequently, evaluating AI-integrated learning must shift from assessing final products to analyzing the strategic nature of human-AI interactions.

Ultimately, generative AI does not inherently democratize learning but amplifies existing disparities in knowledge and interactional competencies. These results highlight the critical importance of educational interventions that explicitly foster both prior knowledge and AI utilization skills.

References

1. Calderhead, J. (1981). Stimulated recall: A method for research on teaching. British Journal of Educational Psychology, 51(2), 211–217. https://doi.org/10.1111/j.2044-8279.1981.tb02474.x
2. Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20(1), 22. https://doi.org/10.1186/s41239-023-00392-8
3. Dochy, F., Segers, M., & Buehl, M. M. (1999). The relation between assessment practices and outcomes of studies: The case of research on prior knowledge. Review of Educational Research, 69(2), 145–186. https://doi.org/10.3102/00346543069002145
4. Hargittai, E. (2002). Second-level digital divide: Differences in people’s online skills. First Monday, 7(4). https://doi.org/10.5210/fm.v7i4.942
5. Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65–94. https://doi.org/10.1007/BF02299613
6. Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10.1145/3313831.3376727
7. Mollick, E. R., & Mollick, L. (2023). Assigning AI: Seven approaches for students, with prompts. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4475995
8. Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). AI literacy: Definition, teaching, evaluation and ethical issues. Proceedings of the Association for Information Science and Technology, 58(1), 504–509. https://doi.org/10.1002/pra2.487
9. Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human interaction with automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055
10. Simon, H. A. (1973). The structure of ill structured problems. Artificial Intelligence, 4(3–4), 181–201. https://doi.org/10.1016/0004-3702(73)90011-8
11. van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2014). The digital divide shifts to differences in usage. New Media & Society, 16(3), 507–526. https://doi.org/10.1177/1461444813487959
12. Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023). Why Johnny can’t prompt: How non-AI experts try (and fail) to design LLM prompts. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–21. https://doi.org/10.1145/3544548.3581388