An Augmented Classroom Teaching System based on AR and Facial Recognition

Objectives

Augmented reality technology (AR) has been widely applied to education in recent years (e.g., Azuma, 1993; Bi et al., 2020). However, most studies on AR in education are limited to providing students with virtual learning resources. Few studies have been conducted to support teachers in enhancing their classroom teaching.

Classroom teaching is a complex activity. Students in the classroom have various backgrounds, behaviors, learning styles, and performances. It is challenging for teachers to manage the learning status of all students in a class and to adjust the teaching arrangement appropriately during the session.

Intelligent techniques, such as AR and facial recognition, can improve teachers’ real-time perception of students’ status. This study explores the application of AR and facial recognition in classroom teaching to enhance teachers’ awareness of students' learning status during ongoing classroom activities. It aims to expand teachers’ ability to monitor student progress or performance in real time during classroom activities.

Theoretical framework

Teaching Augmentation

Teaching augmentation (TA) systems apply various technologies to supplement and broaden teachers' capabilities in classroom activities (An et al., 2020; Holstein et al., 2018b). For instance, real-time learning analytics dashboards have been widely used in the teaching of multiple disciplines by providing real-time information about students’ learning progress and performance (e.g., Molenaar & Knoop-van, 2018; Schewendimann, 2017). Wearable devices such as smartwatches and AR glasses have been employed to support various teaching tasks (Bakker et al., 2013; Berque & Newman, 2015). Additionally, distributed digital lamps have been used to display students’ learning progress and help requests through signals with different colors and pulse rates (Alavi & Dillenbourg, 2012). Despite the development of numerous TA tools to assist classroom teaching, few studies have applied AR facial recognition to aid teachers in classroom monitoring and decision-making.

Learning Analytics

Learning analytics refers to the action of collecting, analyzing, and reporting learners’ data for the purpose of understanding and improving learning (Ferguson, 2012; Lang et al., 2017; Romero & Ventura, 2020). While most learning analytics studies use PCs, mobile phones, or tablets to visualize the analytics data, a few studies (e.g., Holstein, 2018a) have employed AR devices as the interface for data visualization. The results show that real-time analytics have a positive impact on student learning.

Methodology

The proposed augmented teaching system utilizes Microsoft HoloLens 2 and Microsoft Azure Cognitive Services for face recognition, integrated with the Canvas learning management system. Figure 1 illustrates a scenario applying the proposed system in a classroom.

Figure 1

Scenario of Applying AR and Face Recognition Technology in a Classroom

Figure 2 further illustrates the architecture of the proposed system. A teacher wearing a Microsoft HoloLens 2 with the proposed system can gaze at a student and make a ‘click’ gesture to trigger the function, sending a screenshot to Microsoft Azure Cognitive Services for face recognition. Once a student is recognized, the system on the HoloLens 2 retrieves the student’s learning records from a Learning Management System (LMS). Currently, we are using a testing LMS for the prototype development, but we plan to transition to Canvas LMS. Canvas can assist teachers with various tasks such as lesson planning, lecture support, online homework, online tutoring, process evaluation, and teaching management for a variety of teaching modes, including online learning, blended learning, collaborative learning, and flipped classrooms. Figure 3 illustrates the implemented function of the proposed system.

Figure 2

Architecture of the Proposed System

Three research questions were identified: (1) What are the perceived benefits of the proposed facial recognition and augmented reality system on HoloLens for classroom teaching? (2) What are the perceived shortcomings of the proposed systems? (3) What are the concerns to further refine the system?

Figure 3

Implemented Functions to Recognize a Student and Retrieve the Students’ Information

This study used a focus group interview to collect the required information purposefully and systematically. Three university teachers and nine graduates from the Faculty of Education at a university in China participated in the interview. The research purpose and the design of the system were described first. The augmented teaching system was then demonstrated. Participants were encouraged to elaborate on their experiences and to propose suggestions relevant to the research questions. The content of the conversations was recorded during the interview and expanded upon as notes after the interview.

Results

Perceived benefits

Most teachers responded that the proposed system can help them check students’ learning status and then

reflect on the teaching quality. The system has the potential to assist teachers in adjusting their teaching based on student status. Additionally, the system can help teachers become familiar with the students, especially at the beginning stage of a course in a large class. Specifically, the system can avoid awkward situations when a teacher wants to ask a student a question but forgets the name of the student.

Perceived shortcomings

Although most participants had a positive attitude toward the system, some shortcomings were identified. First, the head-mounted AR device Hololens 2 (weighs 566 grams) could make teachers feel fatigued and uncomfortable when worn for a long time. Secondly, the power of the AR device lasted up to 2-3 hours and could not be used for too long at once, so the teacher had to recharge it after a couple of lessons. Thirdly, the appearance of the HoloLens 2 might be a distraction to students. In addition, HoloLens 2 had limited computing capacity (Qualcomm Snapdragon 850, 4GB RAM), making it difficult to use the AR device itself to recognize faces constantly. It took a couple of seconds to get the recognition result from Azure. Finally, it was important to obtain consent from all students in a class considering the system would capture students’ facial information.

Design Concerns

The following ideas were proposed during the interview to improve the design of the system: (1) The currently displayed information (student name, grade level, and learning progress) is limited, and it would be better to provide more information that can support classroom teaching activities, such as randomly showing the name of the student on the AR devices for question answering; (2) Meanwhile, the design of the interface should be clean. Information redundancy could lead to cognitive overload for the teacher if too much information is displayed. We then refined the design of the interface based on the feedback from the participants, as shown in Figure 4.

Figure  4

Refined Design based on the Feedback from the Participants

Discussion

The results of the study align with the previous research on teaching augmentation, indicating that the TA system can help teachers identify students who need support from the teacher (Holstein, 2018a). AR and facial recognition provide a more flexible and intuitive way to identify students and display learning analytics information compared to traditional TA systems on PCs, mobile phones, or tablets. It is feasible to refine the system for wider use in educational practice if the AR device can be improved in terms of weight, size, power, and computing capability.

References

1. Alavi, H. S., & Dillenbourg, P. (2012). An ambient awareness tool for supporting supervised collaborative problem-solving. IEEE Transactions on Learning Technologies, 5(3), 264–274. https://doi.org/10.1109/TLT.2012.7 
2. An, P., Holstein, K., d'Anjou, B., Eggen, B., & Bakker, S. (2020). The TA Framework: Designing real- time teaching augmentation for K-12 classrooms. In P. Bjørn & S. Zhao (Eds.), Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-17). ACM.
3. Azuma, R. (1993). Tracking requirements for augmented reality. Communications of the ACM, 36(7), 50-51.
4. Bakker, S., Hoven, E. van den, & Eggen, B. (2013). FireFlies: Physical peripheral interaction design for the everyday routine of primary school teachers. In S. Jordà & N. Parés (Eds.), Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (pp. 57-64). ACM
5. Berque, D. A., & Newman, J. T. (2015). GlassClass: Exploring the Design, Implementation, and Acceptance of Google Glass in the Classroom. Virtual, In R. Shumaker & S. Lackey (Eds.), Augmented and Mixed Reality: 7th International Conference (pp. 243-250). Springer International Publishing.
6. Bi, J., Liu, C., & Liu, S. (2020). A review of teaching cases of augmented reality based on Hololens. China Computer & Communication, 32(2), 3.
7. Ferguson, R. (2012). Learning analytics: Drivers, developments, and challenges. International Journal of Technology Enhanced Learning, 4(5-6), 304-317. https://doi.org/10.1504/IJTEL.2012.051816 
8. Furlan, R., & Rod. (2016). The future of augmented reality: HoloLens - Microsoft's AR headset shines despite rough edges [resources_tools and toys]. IEEE Spectrum, 53(6), 21-21.
9. Holstein, K., McLaren, B. M., & Aleven, V. (2018a). Student learning benefits of a mixed-reality teacher awareness tool in AI-enhanced classrooms. In C. P. Rosé, R. Martínez-Maldonado, H. Ulrich Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, & B. d. Boulay (Eds.), Artificial Intelligence in Education: 19th International Conference (pp. 154-168). Springer.
10. Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018b). The classroom as a dashboard: Co-designing wearable cognitive augmentation for K-12 teachers. In A. Pardo, K. Bartimote-Aufflick, & G. Lynch (Eds.), Proceedings of the 8th international conference on Learning Analytics and Knowledge (pp. 79-88). ACM.
11. Kharrufa, A., Rix, S., Osadchiy, T., Preston, A., & Olivier, P. (2017). Group Spinner: Recognizing and visualizing learning in the classroom for reflection, communication, and planning. In G. Mark & S. Fussell (Eds.), Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 5556-5567). ACM.
12. Lang, C., Siemens, G., Wise, A., & Gasevic, D. (2017). Handbook of learning analytics. SOLAR.
13. Molenaar, I., & Knoop-van Campen, C. (2018). How Teachers Make Dashboard Information Actionable. IEEE Transactions on Learning Technologies, 12(3), 347-355. https://doi.org/10.1109/TLT.2018.2851585 
14. Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), e1355.
15. Scheeler, M. C., Congdon, M., & Stansbery, S. (2010). Providing immediate feedback to co-teachers through bug-in-ear technology: An effective method of peer coaching in inclusion classrooms. Teacher Education and Special Education, 33(1), 83-96. https://doi.org/10.1177/0888406409357013 
16. Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30- 41.