Evaluating Learners' Online Learning Experience of Informal Learning Environments: A LIWC Analysis

Introduction 

The Internet enables learners to study whenever and wherever and video-based informal learning environments  are becoming increasingly important, even more so after the pandemic (Haavisto, 2021). YouTube does not only  work as an entertainment platform but also as a place where lots of learners gather (Vizcaíno-Verdú et al., 2019).  EdX is a MOOC platform that provides courses from prestigious institutions, and learners visit the website and  watch video-based courses at their own pace. 

Unlike formal learning, informal learning does not have instructors interact with students, control the  pace of content, or identify what learners are understanding. Due to the lack of standardized exams and a low rate  of survey response (Oudeweetering, 2018), it is hard to know much about informal learners' experiences.  Comments are one way to investigate learners' experiences and both environments include discussion features as  part of the interface. For example, YouTube has post-video comments and edX has a discussion forum, which  both work as an asynchronous discussion space for learners (Delello et al., 2019). Since Garrison et al. (1999)  proposed the theoretical framework of community of inquiry (CoI), it has become a solid foundation for  investigating online learning environment. More recently, social network platforms show their potential for  fostering a CoI (Wang & Chen, 2020). In YouTube’s post-video comments, learners share their emotions  (Veletsianos et al., 2018), show gratitude (Lee et al., 2017), and construct knowledge (Dubovi & Tabak, 2020).  Similarly, edX's discussion forum is a place where learners share their experiences, ask questions, and report  issues. These posts contribute to fostering a CoI and the understanding of learners' learning experiences, thus,  helping content producers improve content quality (Liu et al., 2016). 

In this paper, we investigate the similarities and differences between learners’ experience (i.e., CoI) of  two popular video-based informal learning environments containing asynchronous discussion features, YouTube  and edX. This is achieved by comparing the frequency of keywords found in learners’ comments accompanying  educational videos on the two environments and further categorizing them via the constructs of social and  cognitive presence.  

Community of inquiry 

There are three components of CoI: social presence, cognitive presence, and teaching presence (Garrison et al.,  1999). Social presence is the extent that learners can express themselves as individuals in the learning environment and includes expressions of emotion, open communication, and group cohesion. Cognitive presence is the extent  that learners can co-develop meaning through communicating. Teaching presence is how direct instruction,  facilitating discourse, and instructional design facilitate social and cognitive presence. 

Research questions 

YouTube and edX are popular video-based online informal learning environments that likely foster unique CoIs.  Previously, the social and cognitive presence found of these environments has been studied independently by  looking at learners’ comments. To our knowledge, no work has directly compared differences in the CoI  characteristics of these two popular online informal learning environments. Is there evidence of higher cognitive  and social presence to be found in learners’ discussion on the more structured edX than the unstructured YouTube?  Given this, the following research questions guided this study: 

1. How do online informal learning environments foster a CoI? 

2. How do social and cognitive presence differ between unstructured (YouTube) and structured (edX)  informal online environments?  

Based on our research questions, we have two hypotheses:  

1. YouTube’s social presence (as measured by LIWC keywords) should be significantly different from  edX’s. 

2. YouTube’s cognitive presence (as measured by LIWC keywords) should be significantly different  from edX’s. 

Method 

This quantitative study focuses on exploring learners’ discussion and learning experiences in two online  informal learning environments. The data consists of a total sample of n = 2,889 respondents from YouTube and  edX with a total of 6,938 comments. For this research, we chose the math course The Magic of Fibonacci  Numbers from the TED Channel on YouTube. This video has 1 lecture, but 3,204 comments posted by 2,306  learners. In edX, we chose the math course Calculus 1A: Differentiation, published by MIT. It has 42 lectures  with 3,734 comments posted by 583 authors, including one instructor. This research uses the LIWC-22  Dictionary of LIWC analysis (Boyd et al., 2022) to analyze comments’ content, a total of 4,785 comments are  valid for this analysis. 

Two researchers independently categorized each of the LIWC dimensions as either social or cognitive  presence based on the definitions of each CoI element and the keywords from the LIWC dictionary representing  the dimension. The inter-rater reliability of the coding was high, k = .90, and discrepancies were resolved via  discussion. Social presence including the following 6 dimensions: self-identify, affiliation, affect, social process,  culture, and conversational. Cognitive presence included the following 7 dimensions: summary variables, drives,  cognition, affect, states, motives, perception.  

Results 

The frequency of keywords in each dimension can be found in Table 1 and Table 2 and the means reflect their  percentage relative to the total number of words overall. In Table 2, the summary variables from analytical  thinking to authentic are standardized composited variables transformed to a scale from 1 to 100. Independent  sample t-tests were conducted on the means and effect sizes calculated to compare the percentage of each  category between YouTube and edX. For significant differences, the larger of the two means is highlighted in a  color to reflect the size of the effect. The effect sizes are defined, and color coded as follows: d (.01 - .19) = very  small/blue, d (.2 - .49) = small/green, d (.5 - .79) = medium/yellow, and d (.8+) = large/red (Sawilosky, 2009). 

H1. By highlighting significant effects and their relative size in Table 1, differences in social presence  between the environments emerge. Overall, differences range from very small to small. For self-identity words,  edX learners express more personal-centered representations by using more 1st person singular and 1st person  plural words than YouTube’s. EdX comments also have more affiliation words than YouTube's. For affect,  YouTube comments have more positive words while edX comments have more negative ones, except for anger  and swear words. For social processes, edX comments contained more social words including social behavior,  prosocial behavior, politeness, moralization, and communication words, however, YouTube comments have more  interpersonal conflict words, social referents words, and acquaintances words. It seems that edX learners perform  more politely than YouTube learners. YouTube comments garnered more ethnicity words than edX comments,  which may show more diversity characteristics of YouTube. YouTube comments contained more conversational  words, except for assent. As an unstructured informal learning environment, the social presence of YouTube shows slightly more positive affect and culture than edX, which shows slightly more affiliation and social  processes. 

Table 1 

Comparisons between YouTube and edX - Social Presence 

H2. By highlighting significant effects and their relative size in Table 2, differences in cognitive presence between the environments emerge. Overall, differences in cognitive presence range from very small to large (see Table 2). For summary variables, YouTube comments contained more analytical thinking and clout words while edX comments have far more authentic words. EdX learners present more authenticity than YouTube learners. EdX comments contained more drive words such as drive, achievement, and power. For cognition, edX commentshave far more cognitive words, except for all-or-none and certitude. Thus, edX learners show more nuanced argumentation while YouTube comments are more confident and absolute. For affect, YouTube learners use more affect words than edX learners, expressing more positive emotion which also showed in Table 1. EdX comments contained more state words except for fatigue. YouTube commenters use more motive words, except for risk, which shows that edX learners have more sense of risk while YouTube learners engaged with more motivations overall. For perception, edX comments have more perception words, motion words, and space words while YouTube comments contain more attention words, visual words, auditory words and feeling words. As a structured learning environment with more directions and clear learning purposes, edX comments are more authentic, driven, cognitive and demonstrate clear statements. YouTube comments are more motivated, affective, and perceptive.

Table 2

Comparisons between YouTube and edX - Cognitive Presence 

Discussion 

The social presences of edX and YouTube are similar, in general. However, a few differences are interesting to  explore. A higher positive tone and attitude score reflects their choice of a more relaxed approach to learning  (Khan, 2017). This might be influenced by the overall environment of YouTube, an entertainment-oriented  social media platform. In contrast, edX expects students to create a structured learning community in the  discussion, and students are expected to respond to other students’ comments sincerely. Thus, this could explain  the greater level of social processes words found in the edX comments. The cognitive presence of EdX is more  differentiated from YouTube, the comments contain far greater frequency of authentic, drives, cognition, and  states. This could be due to its structured nature, which may promote a higher expectation of cognitive presence  for learners. 

Learners' written comments on the two informal learning environments represent different learning  experiences they had with each environment. By examining the comment data, we identified a more differentiated  cognitive presence in edX and a more similar social presence between YouTube and edX. 

Limitations

The limitations of this research should be considered. First, typos existed in comments, which are not as totally  “clean” data for LIWC analysis. Second, emoji-mediated communication also plays an important role in  comments of informal learning environments, however, this research did not take emoji-based data into  consideration. Future studies could combine the teaching presence of video features and discuss the impact of  teaching presence, while also finding a way to code for the use of emojis. 

Conclusion 

This study revealed that there are some representative characteristics representing the unstructured (YouTube)  and structured (edX) informal online environments in the social presence and cognitive presence of CoI.  YouTube and edX are similar in social presence but different in cognitive presence. As a structured informal  learning environment, edX's curriculum design is more similar to formal learning environments by providing  structured courses, helping students be more authentic, driven, and cognitive. Thus, structured informal online  environments could provide learners with more cognitive perceptions. 

References 

1. Boyd, R. L., Ashokkumar, A., Seraj, S., & Pennebaker, J. W. (2022). The development and psychometric properties of LIWC-22. University of Texas at Austin. 
2. Delello, J. A., McWhorter, R. R., & Lawrence, H. (2019). Fostering a community of inquiry in online discussions.  Academic Exchange Quarterly, 23(1), 51-57. 
3. Dubovi, I., & Tabak, I. (2020). An empirical analysis of knowledge co-construction in YouTube comments. Computers & Education, 156, 103939. 
4. Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher education, 2(2-3), 87-105. 
5. Haavisto, C. (2021). “It’s there to trigger de-learning”: Toward a video-centered pedagogy of difference. Media education on the top (pp. 73-86). Cambridge Scholars Publishing. 
6. Khan, M. L. (2017). Social media engagement: What motivates user participation and consumption on YouTube?.  Computers in human behavior, 66, 236-247. 
7. Lee, C. S., Osop, H., Goh, D. H. L., & Kelni, G. (2017). Making sense of comments on YouTube educational videos: a self-directed learning perspective. Online Information Review, 41(5), 611-625 Liu, Z., Zhang, W., Sun, J., Cheng, H. N., Peng, X., & Liu, S. (2016, September). Emotion and associated topic detection for course comments in a MOOC platform. In 2016 International Conference on Educational  Innovation through Technology (EITT) (pp. 15-19). IEEE. 
8. van de Oudeweetering, K., & Agirdag, O. (2018). Demographic data of MOOC learners: Can alternative survey deliveries improve current understandings?. Computers & Education, 122, 169-178. 
9. Veletsianos, G., Kimmons, R., Larsen, R., Dousay, T. A., & Lowenthal, P. R. (2018). Public comment sentiment on educational videos: Understanding the effects of presenter gender, video format, threading, and moderation on YouTube TED talk comments. PLoS ONE, 13(6), Article e0197331. 
10. Vizcaíno-Verdú, A., Contreras-Pulido, P., & Guzmán-Franco, M. D. (2019). Reading and informal learning trends on YouTube: The booktuber. Comunicar. Media Education Research Journal, 27(1). 
11. Wang, H. C., & Chen, C. W. Y. (2020). Learning English from YouTubers: English L2 learners’ self-regulated language learning on YouTube. Innovation in Language Learning and Teaching, 14(4), 333-346.