L&D 'Transfer Climate'

Training Industry Quarterly (Spring 2008) has an interesting article on L&D 'Return on Investment'.

The title is somewhat confusing 'Take your ROI to Level 6' - given that normally Kirkpatrick is cited as 4 levels. [when expressed as 5 - these are (i) reaction; (ii) knowledge; (iii) behaviour; (iv) impact; & (iv) ROI]

The new sixth Level is proposed (by author Paul Leone of American Express) to be 'Transfer Climate'

This is defined as 'Assessment of factors in the learners' work environment (climate) that will help or hinder the transfer of learning'


This seems very similar to the 'Success Case Method' promoted by Brinkerhoff (see this earlier post on this blog) !

Rather than debate this - I'm more interested in the finding presented from studies at American Express (studying over 2000 managers, attending one of their core leadership programmes).

Leone states "..we found the true impact of a training program will best be predicted by the work climate each participant returns to after the event..."


Key enablers were:

• manager clearly communicates endorsement and support for the training - sets goals and expectations before learner initiates learning event
• manager follows up with participant after the event to discuss what was learned and how to apply
• manager recognizes and rewards improved leadership behaviour

The article highlights that blended learning makes it easier to design formal training that encourages the above - e.g. leader-led kick-off & wrap-up sessions.


What are additional best practises being used by the L&D profession to influence the managers of those attending training courses ?

The Value of Learning (HRD 2008)

At last week's HRD conference I was invited to provide a brief case study in support of the presentation on 'The Value of Learning' given by Martyn Sloman of CIPD.

Last year CIPD, in partnership with the University of Portsmouth, undertook a focused project on 'the value of learning'. This resulted in several tools being created - most notably 'Instrument 3: Establishing the most relevant appropach for your organisation'.

The tool defined four approaches to assessing the learning value contribution:

• Learning function efficiency measures
• Key performance indicators & benchmark measures
• Return on investment measures
• Return on expectations measures

My key points - from our experiences in introducing a global 'bite-size' training curriculum (of 90 minute skill-boosters) - included:

• The tool encourages a systemic, balanced-scorecard approach to evaluating (and more importantly - improving) the value of learning. This ensures consideration of both factors that are important in managing key stakeholders, and those used internally within the L&D team to promote continuous improvement.
• For Learning Function Efficiency measures: we have found it helpful to critically appraise the tutor:delegate ratio of courses ... so for 'skill boosters' running classes with a 1:20 ratio (vs. longer classes with a typical 1:12 ratio). Similarly, working in a global organisation seeking to 'leverage scale' ... there is significant benefits to be negotiated via volume discounts on unit costs (where training classes are provided by an external partner).
• For Key Performance Indicators: in my experience this is a difficult area to establish 'quantifiable' measures directly correlated with L&D activities - however it does drive consideration of whether the L&D approach is supportive of an appropriate psychological contract with employees. 'Bite Size' classes have proved a popular format in helping colleagues reconcile the tension between wanting to invest in class-based self-development, but also needing to stay focused on delivering on current performance goals.
• For Return on Investment measures: I'd suggest that it is probably more rewarding to focus on understanding the extremes (ie why the same class will be highly impactful for some delegates, but not for other colleagues), rather than just measure the 'average' return achieved. Through understanding the enablers & barriers for achieving high business impact attributed to the training class provided, L&D can then act to help more colleagues get more from the course.
• For Return on Expectations measures: As highlighted by CIPD, this is probably the most important of the four types of measures. A colleague of mine introduced me to the phrase 'cheque-book & calendar' - to reflect the tangible signs of senior management support. Hence the fact that senior managers have funded & hosted additional 'bite-size' classes within team and focus-week events (ie outside of the core L&D curriculum) can be seen as a clear sign that there is 'satisfaction' with the value of these courses.

---------------------------

From past articles on this blog on L&D metrics - also see: Reflections on ASTD 2007, part #2

---------------------------

PS - HRD was a great opportunity to network with 'old friend' and meet fellow L&D professionals. More reflections on HRD in coming weeks !

Reflections on ASTD 2007, part #2

A further theme I focused on at ASTD was the perennial challenge of metrics and measures of Learning.

While I wasn't able to attend the Kirkpatrick session, a colleague provided the following insights (thanks Marianne):

• Don't skip any levels - hence don't overlook the importance of 'smile' sheets
• You cannot do meaningful Level 4 evaluations on Leadership programs or team-building (as there are too many other variables affecting the results/ROI). Level 4 is best suited for Sales Training.
• For Level 3 (Behaviour): Survey &/or interview one or more of the following: boss; subordinate; peer; others. Measure before/after if practical.
• It is 'nice' to have a control group, but this is not always practical.
• Get as much reaction as you can in the least amount of time ! - so use a 5 point scale & use a scale with comments as optional.

Robert Brinkerhoff also presented on 'Training Impact Evaluation That Senior Managers Believe and Use: The Success Case Method', based on his recently published book 'Telling Training's Story' (Berrett Koehler 2006).

The main argument here is that a training intervention is likely to have a predictable Impact Distribution.

1. Those who did not try the new skills/knowledge at all
2. Those who tried the new skills/knowledge but reverted back to old ways resulting in little/no benefit
3. Those who tried the new skills/knowledge and achieve positive worthwhile results

So rather than look for the average outcome, seek to understand & communicate what separates out those in category (3).

In doing this it becomes possible to demonstrate that:

• The training intervention CAN work
• The training intervention DOES work when the following factors are in place.

...the latter often being outside the 'control' of the L&D team (eg the actions of the delegate's own manager)

Moreover, if the % that fall into category (c) are determined, and the value of the outcomes they achieved quantified it is easy to illustrate not only the overall ROI, but also to full potential of the training (and all points in-between).

In my view this approach of stepping away from average data, will be very powerful in helping position L&D professionals as consultants: ensuring we have the data to illustrate the responsibilities of leaders, managers and the delegates to achieve the full potential ROI from learning services.