Put Yourself in This Hypothetical Situation: Lee is the manager responsible for health and safety for 1,500 workers in seven plants in the US and Mexico. The company has experienced a problem with injuries caused by a poorly designed conveyor system and non-compliant use of the equipment. Lee decides to arrange online training for half of the workers to test a new online training course. All the trainees pass the final exam at the end of the course. Lee might feel that this was a job well done and go on to train the rest of the workers. But is the job done? Should the other workers be trained the same way? In other words, can Lee be certain the training course was effective?
What is a training effectiveness (impact) study?
Whether you are training one person or the whole workforce, you want to know:
- Did the trainee learn the material? Were the learning objectives met?
- Has the trainee retained the knowledge (information, skills)?
- Has the trainee used the new learning on the job?
- Has the new learning proved useful?
Number one should be answered immediately after the course, with an exam or other means of determining if the trainee understood the key course content/concepts. Along with a course evaluation, this provides valuable information. Lee achieved this. But she stopped there; no further action was taken. This is a mistake because questions 2-4 are even more important.
In many cases, learning is forgotten as soon as the final exam is passed. If that is the case here, Lee paid a significant amount of her training budget for 750 workers to spend 375 hours learning (and then forgetting) important safety information. If the average trainee earns $15 per hour, training time alone cost $5,625. If the course cost $20 per person to deliver, that amounts to $7,500. Even using these low salary and course cost estimates, this training course (if unsuccessful) results in at least $13, 125 being wasted.
An effectiveness study will provide data that can help you know whether:
- you trained the right people
- the training was “sticky” (i.e., trainees were able to recall the learning)
- the training was beneficial
Questions 2-4 should be assessed after some time has passed. This could be after 6 weeks, and typically between 60 or 90 days, but this assessment should not take place more than a year after the training. If you don’t find answers to these questions, you won’t know whether you have trained the wrong group of workers, or if the depth of learning was so shallow that it can no longer be recalled. You won’t know if the learning may be recalled but is not being used appropriately, or if the learning proves not to be as useful as intended.
How does a training effectiveness study work?
Step One: The first step is a brief survey for all trainees to take. The questions on the survey should allow respondents to indicate who did not use the learning at all, who used it with mixed results, and who used it successfully.
The key to the effectiveness results is to find those trainees at both ends of the success continuum (i.e., those who didn’t use the learning at all and those who used it successfully) and find out more from them.
Step Two: The second step is to conduct interviews with a sampling of those at both ends of the success continuum to gather data on their experiences in an effort to determine what happened.
What are the study outcomes?
Study outcomes depend on the questions you ask.
In the initial set of questions, you only need to separate out(a) those who grabbed ahold of the training and used it for their own benefit and for the benefit of others, and (b) those who didn’t remember, didn’t care, or met expected or unexpected barriers they could not overcome.
Whether Step One shows that the training was wildly successful (e.g., it was implemented company-wide and has achieved stunning improvements) or that it was a dismal failure (e.g., no one used the new skills or accidents continued to happen that the training should have eliminated or reduced), you are not done.
Step Two seeks to understand why. Why is at the heart of effectiveness.
Let’s consider one possibility. Let’s say that Lee trained half the workers on proper ergonomic techniques. Three months later, the company is still experiencing similar ergonomic-related injuries. Lee is shocked and concerned. Lee then does an effectiveness study and asks participants:
- Do you remember the training?
- Did the training course indicate any ergonomic adjustments or modifications should be made to the equipment you work with? If so, did you bring it to the attention of your supervisor?
- Did you (or your supervisor) request a follow-up with EH&S regarding the equipment you work with? Did you receive it?
- Are you using the safe lifting and reaching techniques recommended in the training course?
Unfortunately, Lee did not also ask if there is anything that sets any of the trainees apart. In this hypothetical case, it turns out that the half of the workers who were trained worked in the three plants with upgraded conveyance systems. Lee did not train workers in the four plants who worked on legacy systems.
So, of course, there was no change in equipment-related accident rates among the trainees—they were using new equipment with improved ergonomic design. There was also no change among the untrained workers; they continued to develop ergonomic injuries on the poorly designed equipment. If this set of workers had been trained, a significant drop in accidents may have been possible—if they retained and used the learning, and if management supported the use of the learning.
To evaluate the effectiveness of the training, you need to take the time to explore what may have gone right and what may have gone wrong, and get at those answers in the interviews.
You may even want to develop measures to analyze the financial cost of successful training. Your data may show that the training successfully reduced the injury rate, but not enough to justify the expenditure in time and money. Perhaps the money would be more wisely spent on equipment modifications and physical job aids (e.g., posters in the work area) or other awareness campaigns. You don’t know if you don’t ask.
Step Three: The final step involves issuing a report to the company with the training effectiveness study results.
A Few More Training Study Effectiveness Considerations…
- You want honest answers. It is important to phrase questions in a neutral way so as to avoid bias.
- Participants should be notified that no one will be penalized or rewarded due to their responses. The responses are confidential. Only aggregate information will be provided to the company. None of the trainees will be identifiable.
- Trainees may choose to participate or not. While participation in the study is optional, you may want to use small incentives (such as gift cards) to encourage participation.
Evaluating training effectiveness can be a useful tool for demonstrating program return on investment and as a basis to seek future funding for new training initiatives.
Web Design & Development by Endertech