A further theme I focused on at ASTD was the perennial challenge of metrics and measures of Learning.
While I wasn't able to attend the Kirkpatrick session, a colleague provided the following insights (thanks Marianne):
- Don't skip any levels - hence don't overlook the importance of 'smile' sheets
- You cannot do meaningful Level 4 evaluations on Leadership programs or team-building (as there are too many other variables affecting the results/ROI). Level 4 is best suited for Sales Training.
- For Level 3 (Behaviour): Survey &/or interview one or more of the following: boss; subordinate; peer; others. Measure before/after if practical.
- It is 'nice' to have a control group, but this is not always practical.
- Get as much reaction as you can in the least amount of time ! - so use a 5 point scale & use a scale with comments as optional.
Robert Brinkerhoff also presented on 'Training Impact Evaluation That Senior Managers Believe and Use: The Success Case Method', based on his recently published book 'Telling Training's Story' (Berrett Koehler 2006).
The main argument here is that a training intervention is likely to have a predictable Impact Distribution.
- Those who did not try the new skills/knowledge at all
- Those who tried the new skills/knowledge but reverted back to old ways resulting in little/no benefit
- Those who tried the new skills/knowledge and achieve positive worthwhile results
So rather than look for the average outcome, seek to understand & communicate what separates out those in category (3).
In doing this it becomes possible to demonstrate that:
- The training intervention CAN work
- The training intervention DOES work when the following factors are in place.
...the latter often being outside the 'control' of the L&D team (eg the actions of the delegate's own manager)
Moreover, if the % that fall into category (c) are determined, and the value of the outcomes they achieved quantified it is easy to illustrate not only the overall ROI, but also to full potential of the training (and all points in-between).
In my view this approach of stepping away from average data, will be very powerful in helping position L&D professionals as consultants: ensuring we have the data to illustrate the responsibilities of leaders, managers and the delegates to achieve the full potential ROI from learning services.
