Informal learning plays a significant role in overall learning interventions within industry practice. According to the Association of Talent Development, 33% of organizations state that more than half of their offered learning opportunities are informal and only 35% of organizations measure the effectiveness of informal learning (Association for Talent Development, 2024). The same study also found that 51% of organizations believe that informal learning will increase within the next two years.
Formal learning is categorized as structured learning that takes place outside of the working environment and usually within a classroom-like formal education setting (Marsick & Watkins, 1990, 2001). In the workplace, this can look like multi-day in-person workshops that take employees out of their workspace or virtual instructor-led training that is held during the regular working hours of an employee. In contrast, informal learning is not highly structured, and the direction it takes is at the discretion of the learner (Marsick & Watkins, 1990). In the workplace, informal learning manifests as coaching and mentoring programs, communities of practice, self-directed learning through watching videos or listening to podcasts, and learning through trial and error.
Informal learning behaviors have been investigated in the workplace of trainee accountants, graduate trainee engineers, and qualified nurses in work conducted by Eraut et al (2004) to determine how individuals in different contexts were engaging in informal learning in their work environments. A natural extension of this work would be to evaluate the effectiveness of informal learning interventions in the workplace.
Evaluating formal learning interventions can come in the form of pre- and post-intervention surveys, knowledge checks or quizzes within eLearning modules, or focus groups or interviews with participants and facilitators. Due to the fluid and seemingly organic nature of informal learning, conducting structured evaluation practices is a challenge. Since it is learner-driven, pre- and post-surveys are unavailable, and if there are only a few participants at a time, there may not be enough data from interviews to determine the effectiveness of the informal learning intervention.
As the measurement and evaluation team at a large engineering firm, we have begun to develop and implement evaluation methods for formalized learning and propose techniques or adjustments that could be implemented within practice for informal learning interventions. By working within the contexts of current frameworks for learner intervention evaluation for formalized learning, we can adapt them for the informal learning space to more effectively and efficiently deploy learning interventions.
Within the larger context of learning and development at the engineering firm, we have adapted the 4-Levels of the Kirkpatrick Training Model (Kirpatrick & Kirkpatrick, 2016) to 5-Levels including program usage and demographics, participant reaction to the program, learning that occurred as a result of the program, application of content within the program to the participant’s work, and the overall business impact as a result of employee participating in these programs.
To demonstrate a training program’s effectiveness and efficiency through these five levels, we have implemented a 5-step program evaluation process that is adapted from Bagdy and Stefaniak’s Steps for Conducting a Confirmative Evaluation (2025). Each training program follows the evaluation process that defines a problem statement, outlines measurable learning objectives, identifies at least 3 data sources, analyzes data and interprets findings, and communicates to stakeholders. This approach has been effective in helping to demonstrate the business impact or return on investment for the formalized learning programs, which have allowed the business to make data-driven decisions regarding training resourcing and development.
The success of this evaluation approach on formalized learning has led us to consider the informal learning interventions that engineers participate in, aiming to determine their effectiveness and efficiency, and to identify opportunities for optimization to enhance on-the-job application and business impact. Based on Eraut’s (2004) three types of informal learning, it was determined that we begin by focusing on deliberate informal learning or learning that has “a clear work-based goal that leads to learning activities” (Segers, Messmann, and Dochy, 2018). Deliberate learning occurs as part of daily work through problem solving, reflecting on past experiences, and making decisions. Additionally, we determined that we were interested in informal learning through social interaction, or learning from others (Noe et al., 2013), specifically because we wanted to investigate cross-collaboration amongst engineers throughout the organization.
With these considerations in mind, we identified a mentoring program that fulfilled our requirements and had been in operation for several years that we would like to evaluate. The program consists of a cohort of identified mentors who demonstrate the application of organization-established engineering competencies on successful projects and have a company tenure of over five years. At the beginning of the year, engineers with 1-3 years of experience in their role who are interested in being partnered with a mentor can fill out a form to be matched with a mentor on a different project. It is a 6-month commitment that is mentee-driven with no scheduled meetings or kick-offs from the Learning and Development department. The only requested input from mentees is to create a professional goal for the mentorship relationship. Outside of Learning and Development facilitating the mentor match and logging the professional goal, the program is driven by the mentee arranging meetings with the mentor.
Janssens et al. (2017) outlined learning outcomes as changes about a learner, a group, or an organization and refer to several classifications of those outcomes such as affective, or changes in the learner’s motivations or attitudes, cognitive, or changes in a learner’s knowledge, and skill-based, or changes in the learner’s motor or technical skills. For the evaluation work within this program, we have elected to evaluate skill-based outcomes, as they most closely align with the program's deliberate informal learning nature, which supports engineers seeking to advance in their technical field.
After identifying the outcome area that we would like to evaluate, we then adopted the octagon model of informal learning (Decius et al., 2019) to determine areas of measurement concerning feedback, reflection, and intent to learn. Since our mentees are working with a mentor, they will be engaging in direct feedback from the mentor on their progress towards the goal they established at the beginning of the mentorship relationship. After the mentorship program, we would like to evaluate the mentee’s reflection upon completing the program or the subsequent reflection. Moreover, we are interested in the participant’s continued intent to learn as defined by Tannenbaum et al. (2010) as the awareness that the learner needs to improve themselves in the workplace and acquire more work-related knowledge or skills. Specifically, we will be evaluating the learner’s extrinsic intent to learn, or the desire to learn for career growth or development, after completing the program.
In order to evaluate these three areas regarding the skill-based outcomes, we designed three methods of evaluation. First, for direct feedback, we have designed a form that we will request mentors fill out each month of the program, outlining the overall professional goal that the mentee designated at the beginning of the program. The mentor will be requested to select from a list of all competencies that have been internally identified as relevant to the skill set of successful engineers, which they have encouraged the mentee to consider or further develop. And then, it is requested that the mentor write out the guidance that they have given the mentee to continue in pursuit of their professional goal. This information is logged and tracked for the 6-month program to determine which competencies a mentee is working on, as well as the directed feedback that they are receiving to achieve their goal.
At the conclusion of the program, we will conduct interviews and focus groups with mentees to have them reflect on the program and the measures that they took to accomplish the professional goal that they established for themselves. According to Anseel, Lievens, & Schollaert (2006), reflection, in conjunction with feedback processes, can result in information processing shifting from an automatic to a conscious mode, which leads to performance improvements. Encouraging mentees to reflect on their experiences can help them contextualize where they began and where they are as a result of engaging with the program. We will also be asking mentees about their intent to continue their skills development through formal learning opportunities such as internally offered professional development programs and externally offered certification and degree programs.
12 months after the conclusion of the program, we will be evaluating how mentees followed through with their extrinsic intent to learn through determining formal learning opportunities they took advantage of as well as how they have progressed in their career by gathering data on their performance reviews and recording any job title, program, or departmental changes. Evaluating these changes will help us to determine if mentees not only expressed an extrinsic intent to learn, but acted upon in it a manner that resulted in organizational shifts.
At this time, due to human resources constraints on the team’s bandwidth, we are focused primarily on evaluating the outcomes of the mentee. As the evaluation process evolves, we aim to expand our practices to assess the impact of mentoring on an employee's performance. In the future, we aim to investigate whether mentorship in the program influences an employee’s learning intent and their experiences and actions.
Outside of this program, we aim to expand our evaluation of informal learning into more cohort-based programs, where participants collaborate in peer groups to accomplish tasks without structured instruction or within a classroom setting. We are also examining the potential to monitor participants of informal learning to further engage with other informal learning resources or opportunities that are provided through the organization.
EdTech Archives