Learning Evaluation: useless without a strategy

In an interview this week with the UK's TrainingZONE, Martyn Sloman of the CIPD (Chartered Institute for Personnel and Development) made the astonishing assertion that it is not necessary to evaluate training. His statement that “If you’re properly aligned to the business needs and the organization recognizes the value of the training and development there shouldn’t be any need to be obsessed with the figures after the event” is a sentiment we’d all like to agree with, but it’s not realistic.

In order to know that you are continually aligned and to get the organization to clearly perceive your value, you do have to measure the outcomes of your activities on an ongoing basis. Even if you are aligned and if the organization recognizes your value, you still need to keep track of that status in order to be able to adjust your performance and keep those perceptions unshakable.

Now it is true that not a lot of companies bother with evaluation. The American Society for Training and Development does a ‘state of the industry’ study every year.

In 2004, 74 per cent of US companies evaluated training at Kirkpatrick’s level one; 31 per cent at level two; 14 per cent at level three; and only 8 per cent at level four.

In my experience, evaluation within organizations generally is not worth the time and effort that goes into it. Evaluation is generally poorly conceived and executed, often measuring the wrong things in the wrong way, typically subject to significant errors in interpretation, rarely producing actionable or meaningful information, and hardly ever being adequately communicated to decision-makers.

On that basis alone, I’d agree that evaluation, as it is normally carried out, should simply be terminated. OK, keep scaled-back smile sheets as an ego stroke for classroom trainers, but ditch the rest.

Unfortunately companies have reporting requirements, and, like other departments, training departments are under pressure to produce any kind of data which demonstrate that the money being spent has some kind of positive payback. You can’t do that with smile sheets. Nor can you do that with the in-vogue assumption-laden ROI approach, which, to anyone who knows anything about statistics, is fatally flawed at worst, specious at best.

As companies start to accept the changing nature of organizational learning, with an increasing emphasis on the informal, the challenge of evaluation just gets greater. Measuring the impact of formal training interventions is tricky enough, but with informal learning it’s hard to get a grasp of both the impact and the cost, to say nothing of the difficulty of knowing who was even involved.

In putting together evaluation strategies, you have to look at the “desired business impact” end of learning and work back down to the activities that are supposed to cumulatively result in that impact. Typically, trainers seek to do evaluation the other way around: measure as much as you can at the training activity end, then cut and run, leaving someone else to cobble together some aggregate ROI number a year later if it is really demanded.

This leads to the systemic unnecessary collection of huge amounts of irrelevant data with massive disconnects between what is measured and what needs to be measured. The hidden costs of badly-done evaluation are enormous. Trainers and ISDs are simply not qualified to evaluate training, nor should it be a part of their responsibility.

Depending on the training priorities of a company or department, I will usually advocate a strategy that minimizes conventional in-class, all-learners, questionnaire-based data collection, and makes use of well-established market research methods instead.

Use samples instead of evaluating everyone, use bigger collective impacts instead of evaluating every course, and use your results to diagnose problems worthy of more detailed investigation. In order to do this, because you are stepping out of the classroom and into the business, you need to expand the mandate of the training department. This means you need top-level commitment to your evaluation strategy.

You can, for example, get very actionable information about at-work application from a couple of dozen learners in focus groups, rather than having hundreds fill out Likert-laden questionnaires. You can use mystery shopping techniques to confirm actual implementation of, say, customer service skills.

For a very convincing job of demonstrating business impact you can simply “data-mine” already collected information. For instance, in evaluating selling skills training, using small samples of salespeople you can look at before-and-after sales performance and at changes in performance between a control group and a group who has been trained.

Arguing the business case for training is like making a case in court: you assemble the evidence and present it in such a way that it makes a very convincing argument.
You don’t need to produce a smoking gun, and an ROI number can easily be discredited. For those C-level reports in which you justify your ongoing existence, you can select the above plus an array of key performance indicators that reflect, in whole or in part, the impact of the work of your department.

Present your evidence graphically in a dashboard, so that everything is on one page. If you set up your systems to update the dashboard monthly or quarterly so that trends can be established, evaluation can rapidly become an essential ingredient of senior management decision-making.

Far from being unnecessary, evaluation is strategically vital to the ongoing health and success of any training endeavor.