Augmented Reality Design for Literacy Development

The Need

Augmented Reality (AR) refers to technology which superimposes digital elements (such as images, sounds, or text) onto the video or pictorial representation of the real, physical world (Santi et al., 2021), with technological affordances which allow multimodal representation and interaction, personalization, and transfer between virtual and physical world. AR is also widely explored in diverse fields and purposes such as entertainment, architecture, tourism, as well as to support learning (Radu & MacIntyre, 2014). AR was found to support better comprehension and retention of knowledge, physical task performance, collaboration, and motivation across diverse learners and disciplines, such as STEM, language, and visuospatial learning (Dunleavy, 2014; Garzón et al., 2020; Hidayat et al., 2021; Kerawalla et al., 2006; Law & Heintz, 2021; Sommerauer & Müller, 2018); however it may also pose added challenges such as lack of attentional control, difficult usability, and integration into existing curriculum.

While AR-enhanced books may provide added support for literacy development by augmenting book content with alternate presentation (Panchenko et al., 2020) or with interactive activities (Grasset et al., 2008), there is a need to systematically define AR design principles which support diverse literacy development. Literacy is an important and foundational skill developed from the beginning of an educational journey which sets a foundation for academic and lifelong learning. Understanding when and how to implement AR design principles to support certain literacy skills is necessary to guide designers developing learning technologies to support literacy development, educators to select and integrate appropriate tools for their classroom, and researchers to make systematic approaches to inquiry. This study asks: “What are the AR design principles found to be effective in specific literacy development?”

Conceptual framework and methodology

The study conducted a systematic review of empirical literature to identify AR design principles found to be effective in supporting domains of literacy development. Peer reviewed empirical studies were searched using keywords "AR" or "Augmented Reality", combined with "design principles", "language learning", "reading engagement", and/or "systematic review". A total of 24 articles were reviewed to develop a rubric to map AR design principles with literacy development.

AR design principles, or prescriptive, imperative, and context-specific guidelines which enable designers to leverage AR functionalities and affordances to optimize desirable learning outcomes, were categorized into AR affordances: reducing cognitive load; enhancing multi-modal engagement; increasing transfer of knowledge; and increasing positive affect and motivation (Hughes et al., 2011; Sommerauer & Müller, 2018). Literacy was defined as literacy skills, or, the ability to deconstruct and comprehend written and multimodal language consisting of print concepts, phonemic awareness, phonics, fluency, vocabulary, comprehension (NRP; 2000), and literacy behavior consisting of reading motivation and engagement (Baker & Scher, 2002; Coddington & Guthrie, 2009; Davis et al., 2018, p.123; Gottfried, 1990; Rimm-Kaufman et al., 2007; Shapiro, 2011).

Findings

Finding 1: Rubric of AR design principles for literacy development

A rubric (Table 1) was constructed based on a systematic review of AR principles found to be effective in developing reading skills, motivation and engagement. For each literacy development (columns in Table 1), the number of observations where the AR design principle (rows in Table 1) was found to support literacy development was mapped as the number of observations. The rubric provides evidence for AR design principles that are found to be effective in supporting specific literacy development, while identifying the gap in current understanding.

Table 1

Rubric of AR design principles for literacy development

Finding 2: AR affordances for literacy development

Certain literacy skills, behavior, or AR affordances were more frequently observed (Table 1) which serve as evidence to which design principles are effective in supporting a particular literacy development.

Reading interest was most extensively investigated in previous studies followed by vocabulary, value, self-efficacy, and attentional control in the decreasing order of observations. A significant amount of literature has focused on exploring AR design principles to reduce cognitive load in support of reading interest, vocabulary, as well as value, self-efficacy, and attentional control. Furthermore, enhancing multi-modal engagement were found to support reading interest, vocabulary, value, and self-efficacy. AR design principles to increase motivation were found to be supportive of reading interest (M1), vocabulary (S5), value (M2), and self-efficacy (M3). In other words, these design principles not only directly influenced literacy motivation, they also indirectly influenced the learning outcome, specifically vocabulary learning. More traditional literacy skills such as print concepts (S1), phonics (S3), comprehension (S6), depth of engagement (E2), and duration and repetition (E3) were little, if at all, found to be supported through AR. This may also be due to a lack of exploration. What is interesting is there is no differentiation in how certain AR design principles and affordances were integrated for different literacy skills or behavior. In other words, the most commonly observed AR design principles were applied to most commonly supported literacy behaviors and skills.

In terms of AR affordances, reducing cognitive load, increasing motivation, and enhancing multimodal engagement were the most common AR design principles found to be supportive of literacy development. AR affordances to increase personalization and transfer knowledge were relatively unsupported, which may be due to lack of exploration.

Finding 3: AR functionalities for literacy development

23 of 24 studies integrated interactive elements, where a strong correlation was found with interest in literacy behavior. Interactive element derives from detecting user interactions and processing these to generate and position virtual objects within the real-world context for interaction. Enhancing real world visuals was implemented in 19 studies and had a moderate correlation with improved interest in literacy behavior. This functionality recognizes patterns, shapes, or markers to align virtual content with real-world imagery. Sound augmentation was integrated in 5 studies where sound was captured and processed to match visual augmentations. Lastly, synthetic reality display was mentioned in 2 studies, relying on the recognition of images to associate real environments with virtual objects that no longer exist or virtual environments with real objects that were part of an environment which no longer exists. The last two functionalities were minimally supportive of literacy behavior.

Conclusion

Augmented reality (AR) can be effective in addressing some of the known challenges in literacy education, such as personalized support, multimodal and multiple representations, and interactions (Murnane et al, 2012). A systematic review of empirical studies was mapped to a rubric of AR design principles along the categories of literacy development to support educators to determine which AR books to integrate into their classroom based on their design, learning technology designers to implement effective AR design, and for the researchers to explore design principles that are yet examined for literacy support.

We observed a gap in exploring diverse AR design principles for literacy development. Previous studies frequently explored AR to reduce cognitive load. There was little, if any focus on integrating AR designs to support more complex information processing such as personalized learning and transfer of knowledge. This may be due to a bias in how designers and researchers perceive the benefit of AR, with concentration on visual augmentation to guide perception and initial processing of information. Also, previous research mostly focused on increasing literacy motivation (such as increasing interest and value), attentional control, and vocabulary demanding for more wider exploration of diverse AR design principles in relation to literacy skills and behavior as been explored with e-books (López-Escribano et al., 2021). Often AR is discussed for its affordances to support situated learning, connecting the physical world with the virtual representation of knowledge (Dunleavy & Dede, 2014), allowing for deeper information processing, comprehension, and transfer for knowledge. Studies on AR-based literacy development have underexplored such affordances of AR. With known challenges and demands in literacy development where children should learn to construct and transfer complex and multimodal literacies into diverse contexts and affordances of AR to support these needs, limited exploration of AR design limits our current understanding of when and how AR could support literacy development.

References

1. Baker, L., & Scher, D. (2002). Beginning readers’ motivation for reading in relation to parental beliefs and home reading experiences. Reading Psychology, 23 (4), 239–269. https://doi.org/10.1080/713775283  
2. Coddington, C. S., & Guthrie, J. T. (2009). Teacher and student perceptions of boys’ and girls’ reading motivation. Reading Psychology, 30, 225–249. https://doi.org/10.1080/02702710802275371
3. Davis, M. H., Tonks, S. M., Hock, M., Wang, W., & Rodriguez, A. (2018). A review of reading motivation scales. Reading Psychology, 39(2), 121–187. https://doi.org/10.1080/02702711.2017.1400482
4. Dunleavy, M. (2014). Design principles for augmented reality learning. TechTrends, 58(1), 28-34. https://doi.org/10.1007/s11528-013-0717-2 
5. Dunleavy, M., & Dede, C. (2014). Augmented reality teaching and learning. Handbook of research on educational communications and technology, 735-745. http://dx.doi.org/10.1007/978-1-4614-3185-5_59
6. Garzón, J., Baldiris, S., Gutiérrez, J., & Pavón, J. (2020). How do pedagogical approaches affect the impact of augmented reality on education? A meta-analysis and research synthesis. Educational Research Review, 31, 100334. https://doi.org/10.1016/j.edurev.2020.100334
7. Gottfried, A. E. (1990). Academic intrinsic motivation in young elementary school children. Journal of Educational Psychology, 82(3), 525–538. https://doi.org/10.1037//0022-0663.82.3.525
8. Grasset, R., Dunser, A., & Billinghurst, M. (2008, September). The design of a mixed-reality book: Is it still a real book?. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (pp. 99-102). IEEE. https://doi.org/10.1109/ISMAR.2008.4637333
9. Hidayat, H., Sukmawarti, S., & Suwanto, S. (2021). The application of augmented reality in elementary school education. Research, Society and Development, 10(3), e14910312823-e14910312823. https://doi.org/10.33448/rsd-v10i3.12823
10. Kerawalla, L., Luckin, R., Seljeflot, S., & Woolard, A. (2006). “Making it real”: exploring the potential of augmented reality for teaching primary school science. Virtual reality, 10, 163-174. https://doi.org/10.1007/s10055-006-0036-4
11. Law, E. L. C., & Heintz, M. (2021). Augmented reality applications for K-12 education: A systematic review from the usability and user experience perspective. International Journal of Child-Computer Interaction, 30, 100321. https://doi.org/10.1016/j.ijcci.2021.100321
12. López-Escribano, C., Valverde-Montesino, S., & García-Ortega, V. (2021). The impact of e-book reading on young children's emergent literacy skills: An analytical review. International Journal of Environmental Research and Public Health, 18(12), 6510. https://doi.org/10.3390/ijerph18126510
13. Murnane, R., Sawhill, I., & Snow, C. (2012). Literacy challenges for the twenty-first century: Introducing the issue. The Future of Children, 3-15. https://doi.org/10.1353/foc.2012.0013
14. Panchenko, L. F., Vakaliuk, T. A., & Vlasenko, K. V. (2020, November). Augmented reality books: Concepts, typology, tools. CEUR Workshop Proceedings.
15. Radu, I., & MacIntyre, B. (2012, November). Using children's developmental psychology to guide augmented-reality design and usability. In 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 227-236). IEEE. https://doi.org/10.1109/ISMAR.2012.6402561
16. Rimm-Kaufman, S. E., Klein, S. E., & Romberg, M. E. (2007). Social Development Lab-Kindergarten Coding System (SDL-K). Unpublished protocol. University of Virginia.
17. Sommerauer, P., & Müller, O. (2018, June). Augmented reality for teaching and learning- A literature review on theoretical and empirical foundations. Research Papers, 31.
18. Si, Q., Hodges, T. S., & Coleman, J. M. (2022). Multimodal literacies classroom instruction for K-12 students: A review of Research. Literacy Research and Instruction, 61(3), 276–297. https://doi.org/10.1080/19388071.2021.2008555