Many companies do not know if their training is effective. Relying on quizzes at the end of the course may measure short-term memory, but it does not measure understanding, long-term retention, or successful implementation. It also does not measure non-training conditions that influence the effectiveness of training. The absence of more in-depth effectiveness studies is a reason many companies may be blindly throw away money on training.
Here’s the Problem:
The Happy T-Shirt company has a dilemma. With suppliers around the globe, some facilities are experiencing a high volume of slips, trips and falls. These injuries are resulting in lost days, and in some cases, equipment breakage.
Happy T-Shirt company conducts health and safety training for all new employees, and once every five years for refreshers.
The Happy T-Shirt EHS Department charted the injury rates for slips, trips and falls in supplier locations and were alarmed. Compared to the private industry average or about 25 OSHA recordable slip, trip and fall injuries per 10,000 employees, all supplier locations had higher rates, even in the US! And the overall company rate was substantially higher as a result – 32.5 -per 10,000 employees.
Happy T-Shirt Company has 50,000 employees worldwide, so they average about 162 OSHA recordable slip, trip and fall injuries per year. What does this mean to the company?
- The average number of lost work days is 12. (US Bureau of Labor Statistics)
- The estimated average cost per slip, trip and fall injury is $30,000 – 40,000. (National Safety Council).
For Happy T-Shirt company this means (ROI calculation inputs):
Number of slips, trips, and falls (STF) last year: 162
Number of lost workdays owing to STF last year: 1,944
Average fully loaded cost per STF injury last year (lost time plus medical cost): $31,000
Calculated total cost of STF injuries last year: $5,022,000
Fully loaded (training development and training time) cost of STF training is $10 per employee totaling $500,000 for the entire company
Just the Facts:
Happy T-Shirt meets to discuss the findings.
CFO: These injuries are affecting our bottom line.
EHS: Our injury rate is not acceptable. We put controls in place but that’s not enough.
HR: We spent a lot of money on training. That was supposed to solve the problem.
Training: Everyone took the training and passed the quiz at the end.
Labor/Management Committee Chair: We need to find out why the training is not working.
Training is Only Beneficial If:
- The knowledge or skill can change behavior that needs to be changed.
- Training addresses slip-resistant footwear, adequate lighting, unimpeded vision, cordoning off spills before they can be mopped up, etc.
- The behavioral change is designed to solve a specific problem.
- Training addresses the major causes of slips, trips and falls and how to avoid them.
- The training is understood by the trainee.
- Training is in a language and at an educational level that can be understood by the trainee.
- The training is more than checking off a compliance box.
- Training builds motivation to change behavior, not just to go through the motions.
- The training is communicated successfully through good instructional design.
- Training uses approaches, such as interactive activities, that are best for adult learning.
- The knowledge or skill is retained.
- Anyone can retain knowledge for 60-90 days, but is it retained after 9 months to a year?
- The knowledge or skill is implemented.
- Does the trainee use the knowledge or skills as taught consistently and over the long-term?
- The employee’s use of the knowledge or skill is supported by the employee’s supervisor.
- Does the supervisor support and even mentor the trainee in putting the training into practice, or indicate that it probably won’t work or takes too much time?
- The employee’s use of the knowledge or skill is supported by the upper management.
- Does upper management consistently emphasize the importance of employee health and safety and back it up with action?
- The implementation by the trainee has the desired effect.
- Does consistent implementation of training, the trained behavior, reduce the frequency and severity of slips, trips and falls?
- The benefits of the training outweighs the cost.
- Happy T-Shirt company is spending $500,000 on training and yet lost days and the cost of injuries alone are higher than the private industry average. Those training dollars do not seem to be well-spent.
If any one of these elements is missing, there is a good chance that the training will fail to solve the problem.
How Do You Know If Your Training Has the Desired Result?
Effectiveness Studies:
What is a training effectiveness (impact) study?
Whether you are training one person or the whole workforce, you want to know:
- Did the trainee learn the material? Were the learning objectives met?
- Has the trainee retained the knowledge (information, skills)?
- Has the trainee used the new learning on the job?
- Has the new learning proved useful in accomplishing your goals?
Number 1 should be answered immediately after the course, with an exam or other means of determining if the trainee understood the key course content/concepts. Along with a course evaluation, this provides valuable information. Don’t stop there! That is a mistake because questions 2-4 are even more important.
In many cases, learning is forgotten as soon as the final exam is passed. An effectiveness study will provide data that can help you know whether:
- you trained the right people
- the training was “sticky” (i.e., trainees were able to recall the learning)
- the training was beneficial
Questions 2-4 should be assessed after some time has passed.
Often when there is an assessment, it is done 60-90 days after training. While there is benefit to a quick turnaround, you will often miss the more important elements.
- Most people can retain knowledge for 60-90 days, but do they still remember it after 9 months to a year?
- Were all people trained at the same time, or did it happen over the course of weeks or months?
- Did they have the opportunity to use their new knowledge or skills in the amount of time available?
- Has enough time elapsed to determine whether your training goals have been met?
The timing of the effectiveness study should be determined by these types of questions. If you don’t find answers to these questions, you won’t know whether you have trained the wrong group of workers, or if the depth of learning was so shallow that it can no longer be recalled. You won’t know if the learning may be recalled but is not being used appropriately, or if the learning proves not to be as useful as intended. Nine months to one year after training may be a reasonable time.
Effectiveness Study Components
- Pre-Review (optional) — Review proposed training from an instructional design viewpoint. Make recommendations, if any, for improvements prior to training.
- Effectiveness Study Part 1
- Develop survey instrument
- Send survey to select trainees (company to assist by emailing survey recipients in advance requesting participation and sending reminders prior to close of survey.
- Analyze data from participants.
- Submit report of findings with recommendations for improvements.
- Effectiveness Study Part 2 (optional)
- Develop survey interview questions
- Conduct in-depth phone interviews with select survey participants.
- Analyze data from participants.
- Submit report of findings with recommendations for improvements.
- Post-Study (optional)
- Resurvey participants after 6 months – 1 year
- Analyze data from participants.
- Submit report of findings with recommendations for improvements.
The core of an effectiveness study is Part 1. How does it work?
The first step is a brief survey for all trainees to take. The questions on the survey should allow respondents to indicate who did not use the learning at all, who used it with mixed results, and who used it successfully. The timing of the study after the training should take into account whether all training took place at the same time or over a longer period, when trainees may encounter the opportunity to use the training, when results may be observed, etc.
The key to the effectiveness results is to find those trainees at both ends of the success continuum (i.e., those who didn’t use the learning at all and those who used it successfully) and find out more from them.
The survey should be able to pinpoint where problems may be occurring and why: lack of understanding, lack of motivation, lack of support, lack of appropriate controls, etc. Why is at the heart of effectiveness.
The answers to the survey questions may provide you with enough information to determine whether the problems are in the training itself or other influences and to recommend changes to correct them.
There may be a deeper understanding to be found beyond Part 1.
Part 2 is optional. But it may be illuminating
The second step is to conduct interviews with a sampling of those at both ends of the success continuum to gather data on their experiences in an effort to determine more deeply what was successful and what was not. This could take place immediately after the Part 1 report is reviewed.
Return on Investment.
For both the Effectiveness Study Part 1 and the Post-Study, you may want to develop measures to analyze the financial cost of successful training. Your data may show that the training successfully reduced the injury rate, but not enough to justify the expenditure in time and money. Perhaps the money would be more wisely spent on equipment modifications and physical job aids (e.g., posters in the work area) or other awareness campaigns. You don’t know if you don’t ask.
Evaluating training effectiveness can be a useful tool for demonstrating program return on investment and as a basis to seek future funding for new training initiatives.
Happy T-Shirt company decided to call in effectiveness study experts to conduct a Part 1 survey and analysis.
Some of the findings included:
Thorough examination of all the survey results led to changes in training including translation into multiple languages, improvement in controls including lighting, barriers and clean-up of spills, reaffirmation of commitment by management, and reinforcement of important of supervisor support.
After commissioning the Post-Study, Happy T-Shirt company found that improvement had reduced the incidence of slips, trips and falls in all facilities, though not equally. The Return on Investment portion of the study showed that the cost of improved training plus the cost of the effectiveness study resulted in substantial savings in decreased lost time, medical expenses, equipment repair and replacement.
The company was able to reduce their injury rate dramatically from 32.5 per 10,000 to 14 per 10,000 employees, significantly lower that the US private injury average. This translated to 70 OSHA recordable slip, trip and fall injuries after the improved training.
For Happy T-Shirt company this means (ROI outputs after retraining and improvements):
Lost workdays were reduced from 162 to 70
Number of lost workdays owing to STF this year: reduced from 1,944 to 842
Average fully loaded cost per STF injury this year: $32,300
Total cost of STF injuries this year: $2,261,000
Total savings from retraining implementation: $5,022,000 – $2,261,000 = $2,761,000
As a result of the study, the company invested $300,000 to improve barriers, lighting, flooring, and maintenance response to cleaning up liquid on the floor from spills, snowmelt, and other causes.
ROI [savings less cost of retraining, effectiveness study, and improved controls and maintenance] = ($2,761,000 – $840,000)/$840,000 = 229%
The study showed that the return on investment on the improved training and effectiveness studies was significant and substantial.
Happy T-Shirt company decided that in the future they would call in expertise for a Pre-Review of training before it was implemented.
Wondering about your upcoming training?
Phylmar Academy is offering a free Pre-Review assessment on one training topic to the first 10 requests received. Contact us at info@phylmar.com.
Web Design & Development by Endertech