Mystery shopping at Level Three

I have recently been running a training evaluation project with a financial advice company in the midst of the global battle for share of the baby boomer bubble market.

There have never been so many people on the cusp of retirement, and financial advisors are circling them like sharks around a sardine run. The normally staid and sensible financial advertising imagery is giving way to flower-painted VW microbuses, long hair and lava lamps, richly underpinned by the evocative music of the 70’s. We are urged to believe that the person in the dark suit who wants up to 4% of our liquid assets annually in return for helping us to buy a stairway to heaven is just a grown-up hippie at heart, man, who can really relate to our values.

The training challenge is significant if the advisors’ behavior is to synch with the marketing message. Does the repositioning taught in class actually make it through to discussions with potential clients?

The nature of the business makes it very difficult for managers to observe and objectively evaluate those reporting to them. Training effectiveness can be inferred from actual sales results, prospect conversion rates, and before-and-after data mining studies. But if advisors are losing a lot of potential converts, such empirical data do not help to diagnose where (if at all) the training may have been deficient. So how can you be sure that the training is actually leading to on-the-job application?

One solution is to “mystery shop” for financial advice. Using mystery shopping to test customer interaction skills is a widespread approach in industries ranging from cars to cosmetics to coffee. The concept is simple: send someone in to do business and have them report back on the behavior encountered. It is typically used to target and remediate poor behavior in specific sales/service individuals, or to check up on the quality of management of establishments such as restaurants and retail outlets. It is rarely used to evaluate the effectiveness of training. That’s because mystery shopping tends to fall under the control of sales, marketing, or customer service departments who simply assume that training has done its job, and see implementation is a matter of personal choice or supervisor diligence. A direct link between individual on-the-job competence and training is rarely made by such departments, and training departments are reluctant to raise it.

For a wide range of customer contact skills at Level Three, secret shopping can tell us a great deal about the strengths and weaknesses of our training and its impact on behavior. It gives us unbiased feedback, from the perspective of a customer, on how well desired behavior patterns or skills are adopted. It can also tell us a lot about the environmental and systemic obstacles to application of the learning.

Yet trainers don’t often use it as part of their continuous improvement process. There are several reasons:

Conceptual confusion: Beyond Level One, trainers are more accustomed to checking the learner than the learning. The frame of reference is typically testing and opinion surveys, to see how each individual is doing, and such studies are seen to be vast and intrusive. But you don’t need to look at every learner to get a useful indication of the impact of any particular course or curriculum – a small sample will do.

Cost: It’s a lot more expensive to have a shopper talk with an ex-learner than it is to simply e-mail out a link to a Zoomerang questionnaire. Depending on what behavior you need to observe, a mystery shopper can run you anything from $5 to $50 per visit – and that’s in the simple retail customer field where no sophisticated observation is called for. In sophisticated scenarios, such as seeking financial or legal advice, a “shop” may require an hour or more of sustained credibility through complex discussions. You may have to pay qualified shoppers up to $500 per visit, even if they are current staff on your payroll. Add to that the investment in training your shopper squad in observation skills, plus the potential travel expenses involved in getting a national representation in your sample, and the cost of your training QA project can rapidly get into the tens of thousands.

But it need not get higher than that, because you are looking for a diagnostic indication, not a statistically valid sample. You may only visit three or four dozen individuals, the same number you might pull into focus groups. And you may be able get your sponsoring department to help with the costs.

Ethics: Unlike marketers, trainers feel that there is something underhanded, invasive, or unethical in collecting data by subterfuge. But your focus is the training, not the trainee, and the identity of anyone “secretly shopped” should never be divulged to any of his/her management. It may not be retained at all once relevant demographics and training history are appended to the performance observation study.

Liability: Do training departments really want to expose weaknesses in the quality of their training, particularly where each lost sale or disgruntled customer equates to a great deal of lost income? Level Three is the third rail of training evaluation, and most trainers want to stay away from it. The benefits of showing that training produces improved performance may not be significant enough to justify the risk of demonstrating that the opposite is true.

But when the time comes for organizational cut-backs, mystery shopping at Level Three might buy trainers a stairway to heaven; relying on Level One smile sheets may just be a highway to hell.

Can trainers learn fast enough to stay viable?

It seems to me that these days the hard-working trainer gets maligned by just about everyone, including fellow trainers, not for doing a bad job but for not attaining a Renaissance Person status to which few other corporate functions aspire.

I may be more guilty than most of doling out the criticism. I constantly berate those in the training profession for not continuously evolving at the pace of their environment, for being complacently stuck in outmoded paradigms, for defining their roles too narrowly, or for jumping from ineffective low-tech ruts into high-tech ruts that may be equally ineffective.

But when I look at what I am asking of training professionals, I can’t think of any other corporate field in which the desired changes are so broad and so deep:

• Comprehend, master, and stay ahead of strategic implications in emerging technologies.
• Anticipate the direction and performance needs of corporations whose strategic and tactical navigation is in accelerating flux.
• Understand, relate to, and accommodate a wave of digitally savvy employees whose world view, technological competencies, instincts and modes of operation are radically different from those of the established employee base.
• Customize your service to the individual, reducing your operating costs at the same time.
• Continually improve effectiveness, cutting time to market, time to competence, and time away from task.
• Demolish or at least plasticize your formal processes, making them more flexible, more adaptable, and more workflow-snug.
• Develop skills and competencies with constantly evolving tools (personal, group, and enterprise) that span administration, web authoring, testing, evaluation, presentations, databases, scheduling, collaboration, networking, globalization, project management, and communication (broadcast, podcast, mail shot, synchronous, peer-to-peer, mobile).

I could go on, but you get the idea. What about those in other corporate functions? It is true that innovation is called for everywhere, and the impact of e-business touches the goals and processes of all people throughout the organization.

Marketing and sales have gone through significant revolutions in many aspects of their work. Obviously, IT people have different systems to deal with. Customer service people deal online with customers who bought online. Administrative departments have to integrate their operations online with those of suppliers and business partners. Strategy groups are (hopefully) building new visions for the future of the organization. Legal people are rethinking contracts, intellectual property issues, and management of privacy. HR systems are becoming real-time, and more recruiting is done through the internet. And financial people are transacting online.

But I doubt that as individuals there is anyone who has a broader front of continuous change thrust upon them than those with a training responsibility. Nor is there anyone whose fundamental personal operating processes are challenged so deeply. Marketing people may disagree, and trainers have great deal to learn from them about making non-linear change happen rapidly, and about understanding and responding to individual customer needs. But most marketing people specialize in one aspect of the process – training people are expected to be competent across the spectrum.

In most companies, it is not unusual for individual training people to have to set strategic direction, conceive, architect, build, deploy, administer, test, and review everything that they do, with a little help from those in IT. Instructional designers are supposed to help, of course, but too often their only pedagogical qualification is some fluency in Macromedia’s tools – and their ability to provide broad strategic input is limited.

In an environment such as this, there is a tendency to withdraw, redefine our responsibilities within narrow confines, and hope that all that external change will eventually settle down. But it won’t. In a technology sense, and in a workplace culture sense, trainers have to get out more. We have to foster the curiosity and find the time to become more au fait with what is going on in the world of applied technology, e-collaboration, workflow learning, and those aspects of corporate strategy that hinge on knowledge and skills. Trainers need more training themselves, not in task-specific skills but in the environment in which they are going to have to operate.

It is ironic that this “development” part of T&D is the hardest to get budget approval for, yet it is fundamental to the future success of everything we do.

Original in TrainingZONE Parkin Space column of 28 Oct 2005