Development and Validation of a Learner Interactions Behavioral Observation Checklist (BOC)

Introduction

The degree of Learner interactions is fundamental in shaping the quality of learning experiences (Marco-Fondevila et al., 2022). as highlighted by Moore's three types of interaction (2006, 2018). Moore (2018) suggests that learner interactions involve observable relationships among learners, instructors, and content and the cognitive learning processes, ultimately leading to quality learning. Learner interactions are widely used as determinant for learning quality across different delivery environments (Bernard et al., 2009; Tenenbaum et al., 2020). However, data collection approach has heavily relied on self-report, which has raised concerns about bias, timing, and memory accuracy (Fredricks & McColskey, 2012). To address these issues, Bailey, D. (2022) suggested the use of evidence-driven approaches like interviews and observations. Particularly, observations offer a valuable means to collect learner interaction data during active instruction, minimizing subjectivity. Therefore, we developed a Behavioral Observation Checklist (BOC) that offers an evidence-driven approach to gather real-time behavioral data during active instruction. This paper briefly covers the development and validation of the BOC.

Methods

To create a robust tool for collecting learner interaction data, the BOC was developed based on the concept of Moore's three types of interaction (2018). The checklist's items were designed to capture behaviors aligning with the concept of learner-to-learner (L2L), learner-to-instructor (L2I), and learner-to-content (L2C) interactions.

The development and validation of the BOC followed a thorough content validity approach. Four stages were completed in six phases, involving an extensive literature review, item synthesis, refinement, and validation. A total of 17 items (see Table 1) emerged after the first three stages. The validation process engaged seven semi-experts (advanced doctoral students) and twenty experts (experienced researchers) in the field. Each participant was asked to review and complete as directed on an online survey containing 17 items and accompanying open-ended questions.

Table 1

Second consolidation – 17 items: Interactions, Observation items, responses, examples

For the semi- experts’ data analysis procedures, we use the Statistical Package for the Social Sciences (SPSS) and the Aiken V formula (1980), following established criteria from previous studies (Merino-Soto, 2018; Torres-Luque et al., 2018). A critical value of 0.70 at a significance level of p = 0.05 and 0.81 at p = 0.01 were applied to determine whether items should be retained, modified, or eliminated. Items with values below 0.70 were considered for elimination, while those above 0.81 were deemed retainable. Additionally, an effect size analysis, following Merino-Soto's procedure (Merino-Soto, 2018), was conducted using confidence intervals at a 95% confidence level to assess the generalizability of item clarity.

For experts, the analysis also focused on content validity using Lawshe's content validity index (CVI) (1975). The content validity ratio (CVR) was initially calculated, based on experts' judgments of item relevance using a 4-point Likert scale. Items were categorized as either "+1 essential/relevance" (ratings 1 and 2) or "-1 not essential/relevance" (ratings 3 and 4). Our panel of 20 experts aligned with critical ratio value of .49, thus was used to determine whether items should be retained or deleted. Further analyses included calculating content validity indexes (CVIs) at the item-level (I-CVIs) and scale-level (S-CVI) to establish item relevance. The I-CVI indicated the percentage of agreement among experts on each item's relevance, while the S-CVI showed the percentage of relevant items.

Qualitative data analysis for both semi-experts and experts was based on responses from the open-ended questions. Data were analyzed to identify common areas of consensus regarding specific recommendations.

Results

Quantitative analysis for the semi-experts confirmed that all items exceeded the critical value of 0.70, indicating strong alignment with their respective categories (L2L, L2I, L2C). Confidence intervals revealed no significant differences between validation questions for each item, suggesting generalizable clarity (see Table 3).

Table 3 

Three Aiken’s V coefficients for each validation

For the experts, all items were deemed relevant based on content validity ratio (CVR) analysis, surpassing the critical value of .49. Item-level (I-CVI) and scale-level (S-CVI/A) calculations further affirmed item relevance, exceeding 79% and 90%, respectively (see Table 4).

Table 4

Values of Content Validity Index (ne-num of experts indicated essential; n-number of experts)

For qualitative data analysis, semi-experts and experts primarily emphasized the need for terminology clarity and reduction of item overlap. Consideration was given to enhancing the BOC's wording and reducing item redundancy. As a result, a modified version of the BOC was created to better align with the qualitative data feedback.

Conclusion

Behavioral Observation Checklist’s items were found to be valid indicators of learner interactions aligned with the concept of Moore three types of interaction, addressing the need for reliable data collection in the assessment of quality learning and instruction. Most importantly, BOC offers an alternative for collecting real-time behavioral data on learner interactions during active instruction, supporting assessments of quality instructional practices across diverse learning environments. Future research should focus on testing the reliability of BOC in various contexts. Researchers interested in utilizing BOC can contact the authors for access to a modified version of the instrument.

References

1. Aiken, L. R. (1980). Content Validity and Reliability of Single Items or Questionnaires. Educational and Psychological Measurement, 40(4), 955–959. https://doi.org/10.1177/001316448004000419 
2. Bailey, D. (2022). Interactivity during Covid-19: mediation of learner interactions on social presence and expected learning outcome within videoconference EFL courses. Journal of Computers in Education, 9(2), 291–313. https://doi.org/10.1007/s40692-021-00204-w 
3. Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A Meta-Analysis of Three Types of Interaction Treatments in Distance Education. Review of Educational Research, 79(3), 1243–1289. https://doi.org/10.3102/0034654309333844 
4. Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_37 
5. Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575.
6. Marco-Fondevila, M., Rueda-Tomás, M., & Latorre-Martínez, M. P. (2022). Active Participation and Interaction, Key Performance Factors of Face-to-Face Learning. Education Sciences, 12(7), 429. https://doi.org/10.3390/educsci12070429 
7. Merino-Soto, C. (2018). Confidence interval for difference between coefficients of content validity (Aiken’s V): A SPSS syntax.
8. Moore, M. G. (2006). Three types of interactions. In K. Harry, D. Keegan, & M. John (Eds.), Distance education: New perspectives (pp.19-24). Routledge.
9. Moore, M. G. (2018). The theory of transactional distance. In M. G. Moore & Diehl, W. C. (Ed.), Handbook of distance education, 4th edition (pp. 32-46). Routledge.
10. Tenenbaum, H. R., Winstone, N. E., Leman, P. J., & Avery, R. E. (2020). How effective is peer interaction in facilitating learning? A meta-analysis. Journal of Educational Psychology, 112(7), 1303–1319. https://doi.org/10.1037/edu0000436 
11. Torres-Luque, G., Fernández-García, Á. I., Cabello-Manrique, D., Giménez-Egido, J. M., & Ortega-Toro, E. (2018). Design and validation of an observational instrument for the technical-tactical actions in singles tennis. Frontiers in Psychology, 9, 2418. https://doi.org/10.3389/fpsyg.2018.02418