EdTech Archives EdTech Archives The Journal of Applied Instructional Design, 15(2)

Integrating AI in Accelerated Online Bachelor of Business Administration Courses: A Qualitative Content Analysis of the Poe AI Chatbot's Role

Karen Krier, Yu-Hui Ching, & Karen Nicholas

Abstract

This study examines the integration of the Poe AI Chatbots to support non-traditional students in accelerated online business courses. Through qualitative content analysis of student-generated reflections from Business Orientation and International Business Management courses, this research examines how the integration of AI chatbots facilitates the development of research skills, shapes students’ learning experiences, and addresses the unique needs of non-traditional students. Findings reveal that Poe AI Chatbots effectively supported students’ research processes by assisting with generating search terms and identifying resources. However, students also identified limitations in the reliability, personalization, and contextual depth of AI responses. The study underscores the potential of AI chatbots to provide asynchronous, on-demand learning support while highlighting the need for AI literacy instruction and critical evaluation skills. Results suggest that thoughtful pedagogical integration, coupled with explicit usage guidelines and instructor support, can optimize the benefits of AI chatbots for time-constrained adult students in accelerated online formats.

Background

Non-traditional students in accelerated online programs face distinct challenges. Most students work full-time, manage family responsibilities, and pursue education (Tseng et al., 2019; Van Doorn & Van Doorn, 2014) while completing heavier workloads in 7-week courses (Abu-Dawood et al., 2016; Kuo, 2014), risking disengagement or isolation (Roddy et al., 2017; Trespalacios et al., 2021).

AI chatbots have emerged as supportive tools in online learning. Functioning as partners, assistants, and mentors, AI chatbots engage students through dialogue that supports understanding, reflection, and knowledge construction (Goel & Joyner, 2017; Hew et al., 2023; Hyde et al., 2024; Kim et al., 2020; Tamayo et al., 2020; Wollny et al., 2021), providing real-time interaction that helps sustain engagement (Goel & Joyner, 2017; Hyde et al., 2024). Despite these affordances, few studies have examined the use of AI to support non-traditional students in accelerated online courses (Ouyang et al., 2022).

We selected the Poe platform for its free access without requiring paid accounts. Two customized chatbots were developed: one for the Business Orientation course to scaffold research and guide students through online library research, and another for the International Business Management course to serve as a resource partner and support a culminating project. Course-related files and articles were uploaded as the knowledge base, enabling responses grounded in course materials rather than relying solely on general web information.

Purpose and Research Questions

This study investigates how integrating Poe AI Chatbots (Poe) supports students in Business Orientation and International Business Management courses within an AACSB-accredited Bachelor of Business Administration program by guiding students through research, providing resources, facilitating idea generation, and offering asynchronous support. Three research questions guide this investigation: (1) How do student-generated reflections reveal the research process supported by Poe AI Chatbots? (2) How do student-generated reflections demonstrate student experiences using Poe AI Chatbots? (3) How do student-generated reflections indicate the integrated Poe AI Chatbots' alignment with non-traditional students' needs?

Methodology

This study draws on secondary data from accelerated online business courses at a northwestern public university in the United States. This study employs qualitative content analysis (Schreier, 2012) to systematically describe the meaning of qualitative material. Data sources include three sets of discussion board posts, two online library research reflections, and four course reflections from three Business Orientation and one International Business Management sessions.

Following Schreier's approach, we used a concept-driven strategy to construct three main categories from the research questions, then applied a data-driven strategy to derive subcategories inductively from reflections. Inductive open coding involved conceptualizing (identifying recurring concepts), defining (grouping concepts into categories), and developing (arranging categories hierarchically). The unit of analysis was each reflection; the unit of coding was the sentence.

Results

Research Process Supported by Poe AI Chatbots

Reflections from the three Business Orientation course sessions demonstrated how Poe supported the development of research skills. Students indicated that Poe effectively facilitated the generation, expansion, and refinement of search terms. One student explicitly noted, "The AI chatbot provided search terms I wouldn't have thought of on my own, significantly streamlining my research process." Another highlighted Poe's role in providing foundational knowledge: "I received a lot of helpful ideas, the AI Chatbot really helps get a baseline of knowledge about a topic because you can ask basic questions, but also because you can ask questions that further progress your research." In their discussion and reflection, students emphasized Poe’s capacity to suggest research strategies that reduced irrelevant search results, thereby enhancing research effectiveness.

Student Experiences Using Poe AI Chatbots

Student experiences with Poe revealed both affordances and limitations. Students valued Poe for distilling and clarifying complex articles, which helped them understand and annotate key concepts. Nevertheless, students observed quality gaps in the summaries, describing them as “very dry and impersonal" and noting that they sometimes "left out good context." Reliability concerns emerged, as some students worried that the AI "can produce made-up results." They also found some AI responses "a bit generic," prompting requests for "more personalized feedback" and "more detailed responses for course-specific info." Students' prevailing stance reflected cautious optimism, recognizing AI's value while expressing concerns about its reliability. Many described verification behaviors, noting that "you have to double check the sources," while others found that AI-generated summaries confirmed their own thinking and enhanced their confidence.

Students valued Poe for their speed and immediate access to information, but expressed a desire for foundational AI literacy to understand how it works. Even with AI support, students emphasized the reliability of library resources. Some students were curious about comparing different AI tools, while others raised concerns about data privacy. Students articulated a partnership mindset, viewing AI as "meant to work with you and not compete with you," and as a collaborator that complements rather than replaces human thought.

Alignment with Non-Traditional Students' Needs

Students described Poe as a helpful resource for understanding coursework and summarizing sources. In the Microsoft Excel module, students highlighted Poe as "very useful in mastering Excel." Students requested deeper integration and frequently relied on the chatbot for quick questions, platform navigation, and minor technical issues. Students noted that the chatbots' immediacy and on-demand access were well-suited to the demanding schedules of non-traditional students, making it "particularly helpful when [students] need[ed] quick clarification on assignments or course concepts." At the same time, students called for more personalized responses to enhance effectiveness and satisfaction.

Discussion

Student reflections demonstrate Poe's value in supporting research processes among non-traditional students. Educators can leverage AI chatbots as scaffolding tools to help students identify relevant resources and refine research queries, which boosts their confidence in independent research. However, clear guidelines that emphasize the need to evaluate AI-suggested resources are essential for promoting critical evaluation skills.

Students found Poe helpful but recognized the limitations. Concerns about factual reliability and citation accuracy echo findings from existing research (Yang et al., 2025). Students developed conditional knowledge about when and how to use AI effectively, building critical AI literacy. Future implementations should integrate explicit instruction on AI literacy to help students develop skills for evaluating information, maintaining human judgment, and using AI responsibly.

While Poe provided quick clarification and just-in-time support, their responses were sometimes generic and lacked the contextual depth of human feedback. Students preferred greater customization of feedback to better align with their course content and learning needs. Previous research confirms both the potential of AI chatbots to enhance learning and the ongoing challenge of replicating nuanced human interaction (Labadze et al., 2023; Yang et al., 2025). Effective AI chatbot integration depends on balancing response speed, contextual relevance, and human oversight.

Conclusion

Integrating Poe into accelerated online business courses provided meaningful support to non-traditional students, particularly in research and conceptual understanding. However, challenges in reliability, personalization, and trustworthiness highlight the need for careful pedagogical design.

This study did not formally measure academic performance; thus, while students perceived Poe as beneficial to their learning, the direct impact of Poe on learning outcomes remains an open question. Future implementations should incorporate explicit guidelines and targeted AI literacy instruction to optimize the benefits of chatbot use while mitigating potential risks. This study offers evidence on the promise and challenges of AI chatbot integration in accelerated online learning for non-traditional students.

References

  1. Abu-Dawood, S., Barbee, S., Niu, J., West, J., West, T., Cox, L. C., Warren, S. J., & Norris, C. (2016). Evaluating the effectiveness of message design in accelerated online programs using a think-aloud protocol. iConference 2016 Proceedings. https://doi.org/10.9776/16587
  2. Goel, A. K., & Joyner, D. A. (2017). Using AI to teach AI: Lessons from an online AI class. AI Magazine, 38(2), 48–58. https://doi.org/10.1609/aimag.v38i2.2732
  3. Hew, K. F., Huang, W., Du, J., & Jia, C. (2023). Using chatbots to support student goal setting and social presence in fully online activities: Learner engagement and perceptions. Journal of Computing in Higher Education, 35(1), 40–68. https://doi.org/10.1007/s12528-022-09338-x
  4. Hyde, S. J., Busby, A., & Bonner, R. L. (2024). Tools or fools: Are we educating managers or creating tool-dependent robots? Journal of Management Education, 48(4), 708–734. https://doi.org/10.1177/10525629241230357
  5. Kim, J., Merrill, K., Xu, K., & Sellnow, D. D. (2020). My teacher is a machine: Understanding students' perceptions of AI teaching assistants in online education. International Journal of Human–Computer Interaction, 36(20), 1902–1911. https://doi.org/10.1080/10447318.2020.1801227
  6. Kuo, Y. C. (2014). Accelerated online learning: Perceptions of interaction and learning outcomes among African American students. American Journal of Distance Education, 28(4), 241–252. https://doi.org/10.1080/08923647.2014.959334
  7. Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20, 56. https://doi.org/10.1186/s41239-023-00426-1
  8. Ouyang, F., Zheng, L., & Jiao, P. (2022). Artificial intelligence in online higher education: A systematic review of empirical research from 2011 to 2020. Education and Information Technologies, 27(6), 7893–7925. https://doi.org/10.1007/s10639-022-10925-9
  9. Roddy, C., Amiet, D. L., Chung, J., Holt, C., Shaw, L., McKenzie, S., Garivaldis, F., Lodge, J. M., & Mundy, M. E. (2017). Applying best practice online learning, teaching, and support to intensive online environments: An integrative Review. Frontiers in Education, 2, 1–10. https://doi.org/10.3389/feduc.2017.00059
  10. Schreier, M. (2012). Qualitative content analysis in practice. SAGE Publications.
  11. Tamayo, P. A., Herrero, A., Martín, J., Navarro, C., & Tránchez, J. M. (2020). Design of a chatbot as a distance learning assistant. Open Praxis, 12(1), 145. https://doi.org/10.5944/openpraxis.12.1.1063
  12. Trespalacios, J., Snelson, C., Lowenthal, P. R., Uribe-Flórez, L., & Perkins, R. (2021). Community and connectedness in online higher education: A scoping review of the literature. Distance Education, 42(1), 5–21. https://doi.org/10.1080/01587919.2020.1869524
  13. Tseng, H., Yi, X., & Yeh, H. T. (2019). Learning-related soft skills among online business students in higher education: Grade level and managerial role differences in self-regulation, motivation, and social skill. Computers in Human Behavior, 95, 179–186. https://doi.org/10.1016/j.chb.2018.11.035
  14. Van Doorn, J. R., & Van Doorn, J. D. (2014). The quest for knowledge transfer efficacy: Blended teaching, online and in-class, with consideration of learning typologies for non-traditional and traditional students. Frontiers in Psychology, 5, 1–14. https://doi.org/10.3389/fpsyg.2014.00324
  15. Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet? - A systematic literature review on chatbots in education. Frontiers in Artificial Intelligence, 4, 654924. https://doi.org/10.3389/frai.2021.654924
  16. Yang, M., Jiang, S., Li, B., Herman, K., Luo, T., Chappell Moots, S., & Lovett, N. (2025). Analysing nontraditional students' ChatGPT interaction, engagement, self-efficacy and performance: A mixed-methods approach. British Journal of Educational Technology. 56, 1973–2000. https://doi.org/10.1111/bjet.13588