Who says learning should be fun?

At the risk of sounding like a curmudgeon, I am becoming increasingly exasperated by the extent to which “fun” is specified as a requirement of learning design. In fact, it is frequently the only need that is clearly expressed.

Now, I understand the general idea that if people don’t enjoy the training, they are less likely to give it high smile-sheet ratings, or recommend it to a colleague. But are they less likely to learn if the process is not liberally peppered with “fun” experiences?

In theory, we learn best when we are relaxed and in harmony with our learning environment. A good trainer can set the tone and help create the most appropriate atmosphere. Ice-breakers contribute to that state, and are particularly helpful for skeptical learners, those uncomfortable in learning situations, and inexperienced trainers.

But training does not have to be entertaining.

E-learning’s rise has brought this issue to the fore. The constant admonition from instructional designers that e-learning has to be punctuated every couple of minutes with “interactivity” is one of the saddest mantras of our time. It’s like the American notion that food cannot be palatable unless you smother it with ketchup. If you are working with training that is bland and dry, by all means bring on the sauce. But would it not be better to make the training itself more engaging in the first place?

The distinction between engagement and interactivity is crucial, and it’s one that many instructional designers – and those who commission the development work – do not appear to understand. Engagement is intense mental absorption; interactivity is often just busyness or sugar-coating. It is vitally important that learners be engaged. Interactivity, entertainment, and fun can contribute to cognitive engagement. They can equally well distract from it.

There are possibly three causes for the boom in gratuitous fun in learning: managers want motivational experiences disguised as training; instructional designers lack the skills or imagination to architect inherently engaging learning experiences; and trainers seek high smile-sheet scores and possibly a release from their own boredom.

Managers: Too often, when managers commission training, what they really want is a motivation activity. I can't tell you how often, for instance, I have heard the whine: "We can't make people take that course online, because we'd lose out on the motivation, energizing, bonding, or social interaction that we get from the classroom course."

My response is to challenge the learning objectives: if the primary desired output of the "event" is all that warm fuzzy stuff, then why pretend to be running a training course? It is far more cost effective to structure a motivation session that achieves those goals, and does not come out of the training budget. (Through my focus on sales and marketing, I have designed and run many such events, some of them wild or lavish, and all of them very effective, but I have never called them "training" events). Naturally, if the self-discovery or bonding is germane to the learning (in, say, leadership training or some interpersonal skills training) then it may be relevant and appropriate.

But too often the kind of feel-good factor that is called for in training is (once again) training being abused as a surrogate for good management.

Instructional designers: Many learning experiences are atrociously conceived. The designer has to work with unrealistic timelines, limited subject matter knowledge, poorly specified learning objectives, and myopic supervision, as well as limitations or constraints in the way the training is to be implemented.

The expedient approach often taken is a linear exposition of content, made less dull by the frequent insertion of bits of fun. “We’ll do an icebreaker at the start, a game here, bring out the Lego over there, run a video here, chuck in some role-playing and a bit of team competition intermittently, and make the tests like Who Wants to be a Millionaire. ‘Triple Bottom Line Accounting for HR Professionals’ will never have been so much fun! Who cares if a week later the only thing they remember is the egg-dropping contest?”

Instructional designers need to look beyond the smile sheet for their inspiration, and companies need to stop hiring as ISD's people whose only pedagogical qualification is a mechanical competence in Macromedia's tools.

Trainers: From a trainer's perspective, "having fun" can be an acceptable way to leaven the densest of subject matter, so long as it does not distract from the learning goals. And, because in too many companies the smile sheet ratings are the primary indicator of how good a trainer is at his/her job, "fun" is welcome. It is often true that a learning experience becomes less engaging for the trainer the more engaging it is for the learner. If you are running the same course over and over again, the entertaining components serve to keep you from putting yourself to sleep.

From a learner's perspective, those who do not really want to participate in the learning experience in the first place may be distracted, if not seduced, by the injection of "fun" components; those who really want to learn ask themselves why they are wasting time on irrelevant padding.

Fun can be very constructive. Well-designed entertaining experiences that are relevant and fully integrated into the learning process can work as powerful illustrations of concepts or living analogies. If the fun is designed as an effective instructional process, contributing to the achievement of specific learning objectives, I'll opt for fun over dull any day. But if it is gratuitous, I won't waste my time, or that of learners, by indulging in it.


Original in TrainingZONE Parkin Space column of 21 Oct 2005

A child's view of tomorrow's learning

The US Departments of Commerce and Education recently commissioned a study of more than 160,000 school-goers from kindergarten to Grade 12 to explore their views on technology for learning. The report has just been released, and it makes compelling reading.
Why do the views of these Newmils matter? They see the world, and interact with it, differently from earlier generations. Today’s US K-12 pupils were born into digital technologies. They intuitively integrate things like computers, the web, instant messaging, cell phones, and e-mail into their daily lives.

I’ve not seen more recent data, but way back in 2002 children aged 13-17 were already spending more time accessing digital media than they did watching television. For the US, that was a huge milestone. (Of course, TV is a digital technology these days, so there goes another useful trend line.) By 2002, every classroom in every public school had internet access, and nine out of ten 5–17-year-olds used computers. According to their parents, more than a third of children ages 2-5 went on-line, up from only 6% in 2000. By now it is safe to say that the internet and other digital technologies are ubiquitous for the youth of America.

The study asked this digitally savvy group what they would like to see invented that would help kids learn in the future. Then the authors of the study consolidated the results and came up with a description of the vision that school-goers have for the future.
I’ll quote the report verbatim:

Every student would use a small, handheld wireless computer that is voice activated. The computer would offer high-speed access to a kid-friendly Internet, populated with websites that are safe, designed specifically for use by students, with no pop-up ads. Using this device, students would complete most of their in-school work and homework, as well as take online classes both at school and at home. Students would use the small computer to play mathematics-learning games and read interactive e-textbooks. In completing their schoolwork, students would work closely and routinely with an intelligent digital tutor, and tap a knowledge utility to obtain factual answers to questions they pose. In their history studies, students could participate in 3-D virtual reality-based historic re-enactments.

Now to me this sounds bland and unimaginative, almost status quo, not at all the kind of creative or exciting thing that kids should be coming up with. It sounds like something a committee of adults would produce one evening over tea and biscuits. And indeed, that is effectively what seems to have happened. All of the spontaneous innovative ideas produced by 160,000 kids were filtered, sanitized, and compromised by a committee of "analysts" who stripped all of the freshness out of them and boiled them down into this pedestrian, adult, government departmental interpretation of a child’s vision.

The authors admit to throwing out all but 8,000 responses, and then only looking at those that met their pre-determined criteria of "meaningfulness". Beyond the summary, they do provide a few actual examples of real responses to illustrate their conclusions and in those carefully selected comments lie some clues to the gold that was not mined.

There were numerous requests for pocket-sized multi-functional computers linked wirelessly to the web, pre-loaded with text books. The next evolution of the iPod Nano perhaps, or an extension of the iTunes-web-enabled mobile phone.

The kids in this study talked about wanting automated learning, straight to the mind, using teaching-hats or smart helmets, cable connections in the head, or wireless chips implanted in the brain. I remember as a child wishing that I could put on headphones before going to sleep, flip a switch on a cassette player, and wake up the next day with all of my schoolwork already remembered. It seems that is an enduring desire in the species, only the technology gets updated. Learning is like losing weight and getting fit: we all want the end result, but the process we have to go through is so unpleasant that many of us will avoid it if we can.

While this report appears to be written by those who only see what they want to see and can only understand that which fits their current frame of reference, I may be wrong. If the responses are really as broadly unimaginative as they are represented to be, maybe that’s just further evidence of the challenge faced by technology developers: even kids can’t visualize what they can’t conceive; they really do not know what they need or want until they see it.

I hope that the original responses to this survey have not been discarded and will be made available for other analysts. You can download a copy here.

Blackboard + WebCT = swan song for the LMS industry

So Blackboard is to acquire (sorry, merge with) WebCT. Does it mean anything, other than to all those folks who will soon be looking for jobs?

What we are seeing is simply further implosion of the dedicated e-learning technology industry. The more oligopolistic this market becomes, the more generic it becomes, and the less able it is to sustain the pretense of any meaningful differential advantage. As open source systems undermine it from below (particularly in the academic arena) and ERP systems make it redundant from above (particularly in the corporate arena), the less relevant this relatively small software market segment becomes.

Hence the increasing investor wariness. Soon to be followed by more enthusiastic uptake of what I have been advocating for ten years now – educators, trainers, and especially learners will start to focus less on the means and more on the end, invoking whatever technologies happen to be mainstream to facilitate whatever learning experience is most appropriate to them. IT departments will find it easier to wrest away from training departments the decisions about enabling technologies, and learning information flows will move out of their relatively proprietary niche and finally become fully integrated with the rest of the corporate nervous system.

The openness and dynamism of the web will finally be allowed to permeate the thinking of the learning establishment, and Model-T e-learning will succumb to a flood of performance-driven innovation.

Learning from vaporware: how small is the future?

Small is the new big. Nanotechnology is the next big thing. Micro-payment transactions proliferate. From the business-card sized iPod Nano to the 100-minute bible, everything is being reduced to smaller or faster objects of consumption. This should be familiar to trainers who for years have been under pressure to reduce course length, cut times to competency, and walk on water. But some things can’t get smaller: you can shrink a Wurlitzer jukebox to something you could slip in your wallet, but you can’t make the songs it plays any shorter.

Can traditional learning experiences be decimated and served up in nano-chunks without losing effectiveness? Or might they actually be enhanced? Unlike music, where the song is the focus, learning is all about impact on performance – the ‘course’ itself doesn’t matter. If we are to take custom-tailoring of learning experiences to heart, the more granular our solutions, the more accurately we can fit each learner’s individual needs.

Smallness is one of the key characteristics of the content of Internet 2.0 and, in turn, of what is becoming known as “e-learning 2.0”. Big chunks of solidified content are simply not easy to look into or combine. With sandstone blocks you can build pyramids; with sand you can build anything from beaches to hourglasses.

Perhaps one of the greatest shortcomings of the early web and the applications that have to date been enabled by it is that it has primarily been about recording, organizing, and making accessible "stuff" from the past. There's great value in that. But the necessary emphasis on the here-and-now, and on the future, is rapidly coming into focus. It's not content, or even context, but process that gets us where we are going. We are not what we have done, but what we are trying to do. All of the diverse components of e-learning 2.0 aim to fill the vast white spaces between the legacy content piles with dynamic processes that can instantly mine the relevant diamonds from those piles, and pull them together in real-time into something unique and immediately useful. Some of those processes are lively discussion, synergistic collaboration, spontaneous project work, and nano-customization. Just as micro-transactions are starting to make e-commerce really different from traditional commerce, micro-learning experiences will make e-learning 2.0 really different from Model-T e-learning. Or that’s the theory.

Smallness is increasingly important in all data flows, and learning is simply another kind of dataflow. If learning is water, old-school SCORM learning objects are ice-cubes: uniform, predictable and transportable, but they melt and lose their usefulness rapidly. What we really need to do is vaporize the water. Knowledge vapor is simply learning liberated, in its smallest possible components – unlike learning objects, if you do not contain it, it disseminates itself far and wide, except where the circumstances are created to condense it back into liquid, or ice, again. The best technology available right now for vaporizing and liberating learning and for finding, filtering, and condensing it as needed, is the human mind. Of course lots of cool web technologies are emerging to help facilitate that process (e-mail was probably the first; blogs and wikis are getting there, as are communities of practice; mobile broadband helps). But technology is not the real key to the success of e-learning 2.0. What connects all our small pieces of learning is not a technology, but our humanity.

As far as jargon goes, I must confess to having major objections to the term "2.0" as applied to the web and to e-learning. It suggests a formal release of a new beta-tested version under some kind of planned production process. That's not the way things work any more. The evolution of technology and our ability and willingness to use it creatively has outpaced our ability to manage it, at least in any traditional sense of the word "manage." Trying to "manage" internet-enabled progress is increasingly delusional, like King Canute trying to command the tide not to come in. Agility, opportunism, plasticity, and instinct need to replace outmoded notions of structure, hierarchy, and traditional planning and financial controls, not to mention the sacred cow of centralized "managed" corporate learning.

But since (largely thanks to stock markets, tax authorities, and standardized accounting practices) our current economies rest on those rotting bureaucratic timbers, the bigger corporations may not be able to change, to take the right actions, in time to stay relevant and save themselves.

Google rose from nothing to become the new Microsoft so rapidly it looked like a conjuring trick. China makes 80% of everything sold by Wal-Mart, the world’s largest retailer, a transformation that took less than five years. Before the ink was dry on most American e-learning IPO’s, Indian companies were eating their lunch. It may be only months before the next new megacorp bursts from the web and makes even Google look tired and outdated. What role will training play in the success, or failure, of those endeavors? Can you train an organization to be agile, instinctive, anticipatory, and adaptive? Or can we merely work to remove the containment of knowledge and facilitate the vaporization of learning?

Best practices in questionnaire design

Recently so many people have been asking me to review their questionnaires and surveys that I thought I’d update a document I first created several years ago which sets out some essential best practices for creating good questionnaires. While written for training evaluation, the guide is applicable to any surveys.

1. Ask: “Why are we doing this?”

• What do we need to know?
• Why do we need to know it?
• What do we hope to do when we find out?
• What are the objectives of the survey?

2. Ask: “What are we measuring?”
In training evaluation, what you measure can be influenced by the learning objectives of the course or curriculum you are measuring:

• Knowledge
• Skills
• Attitudes
• Intentions
• Behaviours
• Performance
• Perceptions of any of the above

Your questions, and possibly your survey methods, will differ accordingly.

3. Be aware of respondent limitations.

• Where possible, pilot your questionnaire with a sub-group of your target audience.
• The complexity of your questionnaire and its language should take into account the age, education, competence, culture, and language abilities of respondents.

4. Guarantee anonymity or confidentiality.

• Confidentiality lets you follow up with non-responders, and match pre- and post studies.
• Confidentiality must be guaranteed within a stated policy.
• Anonymity prevents you from doing follow-ups or pre-post studies.

5. Select a data collection method that is appropriate.
Consider the speed and timing of your study, the complexity and nature of what you are measuring, and the willingness of respondents to make time for you. Options:

• E-mail – fast, inexpensive, not anonymous, requires all respondents have e-mail.
• Telephone – time consuming, not anonymous, may require skill, has to be short.
• Face-to-face interview – slow, expensive, requires skill, best for small samples, qualitative studies.
• Web-based – fast, inexpensive (if you use services like Zoomerang), can be anonymous, best for large surveys.

6. Write a compelling cover note.
Where appropriate introduce your questionnaire with a brief but compelling cover note that clarifies:

• The purpose of study and why it is worth giving time to.
• The sponsor or authority behind it.
• Why you value the respondent’s input.
• The confidentiality or anonymity of the study.
• The deadline for completion.
• How to get clarification if necessary.
• A personal “thank you” for participating.
• The signature or e-mail signature of the survey manager (or, ideally, of the sponsor).
• If sending an e-mail, have it come from someone in authority who will be recognised, use a strong subject line that cannot easily be ignored, and time it to arrive early in the week.

7. Explain how to return responses.
If not obvious, make it clear how and by when responses must be returned.

8. Put a heading on the questionnaire.
State simply what the purpose is, what the study is about, and who is running it.

9. Keep it short.

• State how long completion should take and make sure that it does.
• Make questionnaires as brief as possible within the time and attention constraints of your respondents (personal interviews can go longer than self-completion studies).
• Avoid asking questions that deviate from your survey purpose.
• Avoid nice-to-know questions that will not lead to actionable data.

10. Use logical structure.

• Group questions by topic.
• Grouping questions by type can get boring and cause respondents to skim through.
• Number every question.
• Where possible, in web-based surveys put all questions on one screen, or allow respondents to skip ahead and back track.

11. Start with engaging questions.
Many questionnaires are abandoned after the respondent answers the first few questions.

• Try to make the first questions non-intimidating, easy, and engaging, to pull the respondent into the body of the piece.
• Try to start with an open question that calls for a very short answer, and ties in to the purpose of the questionnaire.

12. Explain what to do.
Provide simple instructions, if not obvious, on how to complete a section or how to answer questions (circle the number, put a check mark in the box, click the button etc.)

13. Use simple language.

• Avoid buzz words and acronyms.
• Use simple sentences to avoid ambiguity or confusion.
• If necessary, provide definitions and context for a question.
  

14. Place important questions at the beginning.

• If a question requires thought or should not be hurried, put it at the beginning. Respondents often rush through later questions.
• Leave non-critical or off-topic questions, such as demographics, to the end.

15. Select scales for responses.

• Keep response options simple.
• Use scales that provide useable granularity.
• Make response options meaningful to respondents.
• Make it obvious if open-ended responses should be brief or substantial by using an appropriate answer-box size.

16. Fine-tune questions and answer options.

• Keep response options consistent where possible - don’t use a 5-point scale in one question and a 7-point in the next unless absolutely necessary; don’t put negative options on the left in one question and on the right in another.
• Be precise and specific – avoid words that have fuzzy meanings (“rarely” or “often” or “recently”).
• Do not overlap response options (use 11-20 and 21-30, not 10-20 and 20-30).
• If you use a continuum scale with numbers for answer options, use a clear concept at the top and bottom of the scale (instead of “on a scale of 1 to 5, how good is it? : 1-2-3-4-5, use 1=very bad -2-3-4-5=very good).
• Use scales that are centred– don’t have one “bad” answer option and four shades of “good”.
• Don’t force respondents into either/or answers if a neutral position is possible
• Allow for “not applicable” or “don’t know” responses.
• Edit and proofread to make sure that answer choices flow naturally from the question.

17. Avoid leading or ambiguous questions.

• Don’t sequence your questions to lead respondents to answer in a certain way.
• Avoid questions that contain too much detail or may force respondents to answer “yes” to one part while wanting to answer “no” to another (e.g. “How confident do you feel singing and dancing?”).
• Minimise bias by piloting your questionnaire before it goes live.

18. Use open-ended questions with care.

• Open responses are difficult to consolidate, so use them sparingly.
• They often provide really useful data, so don’t avoid them completely.
• Doing a pilot or running a focus group before rolling out a survey can provide useful insight for creating more structured closed questions.
• Provide at least one open question so respondents can express what is important to them.

19. Thank the respondent.

• Thank the respondent once again. Reiterate why you value the input.
• If you intend to feed back results, emphasize when and how they can expect to get them.
• If you have offered an incentive, specify what the respondent has to do to claim or be eligible for it.


Original in TrainingZONE Parkin Space column of 2 September 2003

Learner-created learning

I have always been convinced that one key to future success in web-based learning lies in the notion of the prosumer. Prosumers produce what they consume, and it seemed to me back in the mid 1990’s that creating their own content was something that people really enjoyed doing. I saw this in the fact that e-mail was the most-used application of the Internet. I saw it in the fact that AOL members spent more online time in chat rooms than in any other activity. I saw it in the enormous popularity of online massively multiplayer role-playing games, where, though the environment is provided for them, players create their own unique characters and pursue their own adventures. And I saw it in the early rise to prominence of special interest online communities such as Parent Soup where, much like trdev today, members created content for each other, with some editorial guidance.

In the early days of web-based learning, before the oppressive influence of standardized Learning Management Systems, before SCORM took the spark out of ISD, in amongst the prevailing stand-alone conversions from CD-ROM there was a lot of interesting experimental design going on. Much of that innovative learning design centered on using the web for what it did best – connecting people with people to share experience.

When I started an e-learning company in 1998 to provide a project management curriculum online, I built community into the design, rather than simply replicating the tried-and-true classroom versions of the courses. While the content of the course was delivered in a relatively traditional way, it was structured to have learners collaborate with each other, creating their own supplementary content. Every learner had a SME mentor, who was available by e-mail throughout the course. The pool of SMEs hosted online chat sessions around the clock, covering topics related to course content, where learners could exchange ideas, trade war stories, and get clarification on issues. Those sessions were all logged, scrubbed of proper names, and stored in a searchable online library. After tens of thousands of learners, that organically growing repository of community experience was a fabulous resource. The community was so valued by learners that many subscribed to it after completing their courses, so that they could continue to engage with each other and access the content.

That booming company was acquired by a traditional learning business that had no time for such esoteric notions, and stripped the courses back to computer-pumps-it-at-you mode. They saw e-learning as a way to cut costs even at the expense of dumbing down learning effectiveness, and providing human interaction was considered counterproductive. Whether it was the result of ignorance, tunnel vision, technology standardization, or accountants getting the numbers wrong, such was the fate of most e-learning around the beginning of this century.

But the prosumer market is still with us in other fields, and it is stronger than ever. Blogging is the obvious example, and it is till growing so fast that the statistics are out of date the moment they are published. Along with creating blogs, publishing your own photographs on the web has taken off, aided by free photo-hosting services like Flickr. Communities such as Del.icio.us, which are specifically designed for sharing information and links of mutual interest, are booming. And services like 43Things and Backpack, which help you tie all of these together, are starting to take off.

In training, we are seeing prosumer concepts like informal learning, workflow learning, and collaborative learning coming into vogue. These are still considered by mainstream learning professionals as interesting but impractical, largely because they are hard to conceptualize and harder still to manage. Yet there are so many reasons why we should be dedicating at least some of our resources to experimenting with them. One reason is that there are lots of technologies out there that, with a little imagination, could be used to make collaborative learning more practical. Another reason, the most important, is that people have demonstrated time and again that they like to interact with others, and that they find creating their own content motivating and compelling.

In the computer games industry, the conventional wisdom is that online games that have been massively successful have all allowed players to substantially influence their environment and leave their mark. Much as in the real world of clubs and associations, loyalty is sealed if the participant has invested time, energy, and creativity in building a presence that others can interact with and appreciate.

It’s time we stopped treating e-learners like members of an anonymous audience in a darkened theatre, and started inviting them all up on stage.


Original in TrainingZONE Parkin Space column of 26 August 2005

Evaluation professionals are undervalued

The evaluation of training is too important to be left to trainers.

Unless certification is involved, at the individual intervention level, at the strategic enterprise level, and at all points in between, the quality assurance processes applied to formal learning initiatives in most organizations are often rudimentary at best.

That does not mean that the quality of training is poor, just that we have no real data to support our feeling that we are doing a good job. Training departments are usually stretched thin, and don’t have the time or resources to do a “proper” quality assurance job at either the course level or at the aggregate departmental level.

Informal evaluation is done by the trainer, most of who know whether an activity is “going well” or not. Formal feedback is carried out at course end, and emphasizes learner reactions, likes and dislikes. And, if the activity involves testing, there are always the scores to look at.

These are all important elements in assessing the quality of a learning experience, and they provide valuable feedback to a trainer. But they are not enough, not by a long shot.

When left in the hands of trainers and instructional designers, the focus of evaluation is too micro, too inward looking. The purpose of training is to improve organizational performance through improving the performance of individuals and teams. Learning evaluation should serve that purpose, as a quality assurance tool. To do that, evaluation has to be pan-curricular, and must adopt a higher level perspective.

This “helicopter view” is hard to achieve if the responsibility for designing and implementing evaluation is too course-specific. Yet who has the time, or the mandate, to step back from a busy course development or training schedule and get strategic?

Only the largest firms have dedicated evaluation resources who know what they are doing and have the credibility to influence policy. And even those resources are becoming imperiled by the inroads that the LMS is making.

Does it matter? There are several reasons why it does.

Implementing a regimen that elevates the strategic importance of evaluation (across all levels) and places it on a more professional level will do two vital things. It will improve significantly the effectiveness and efficiency of all learning activities; and it will save a tremendous amount of unnecessary, un-useful, or redundant work.

My fear is that with the advent of LMS-based evaluation and record-keeping, the information we have about the quality of our learning activities is becoming more narrowly focused, and its usefulness is becoming further diluted. Just as LMS functionality tends to constrain the nature of our design of instruction, it constrains the nature of our inquiry into its impact.

I’d like to see more training departments creating evaluation units and staffing them with a trained expert or two who can help get past the simplistic "smile-sheet & ROI" approach and start building systems that put the important issues on the dashboards of individual trainers, instructional designers, and senior learning managers.

Some LMS tools claim to be able to do just that. But, as with all tools, without a trained and committed hand to guide them, they simply don’t get used.

Just as most of us never use more than 5% of the potential of our spreadsheet software, the potential of emerging tools does not get realized. Automation was supposed to help us do things better; in reality, it often makes us complacent. We dumb down our expectations, dumb down our evaluations, and ultimately dumb down the business impact of our training endeavors.

The “hot career” of the past five years was Instructional Systems Design. I’d like to see companies valuing Learning Evaluation professionals as highly. They can contribute substantially to the quality of training, and to the business results that it achieves.


Original in TrainingZONE Parkin Space column of 19 August 2005

Revisiting Kirkpatrick's Level One

Whenever I am involved in an evaluation project, I advocate getting rid of the smile sheet completely, and replacing that tortured questionnaire with one closed question, plus an open follow-up to encourage respondents to reveal what really matters to them: “Would you recommend this course to a friend or colleague? Why or why not?”

The response tells you unambiguously about the level of satisfaction of the learner, and any clarification offered tells you about the issues that really matter to that learner. That’s more than is called for at Level 1, especially if you have done a good job of testing your training intervention before rolling it out live.

It’s not always possible to reduce things to one question, but I see it as a starting point in the negotiation. I tend to be somewhat dismissive of Level 1 evaluations. That is not because they serve no purpose (they are vital), but because they attract way too much attention at the expense of business impact studies, and because they are often poorly designed and inaccurately interpreted.

Every training intervention needs some kind of feedback loop, to make sure that – within the context of the learning objectives – it is relevant, appropriately designed, and competently executed.
At Level 1 the intention is not to measure if, or to what extent, learning took place (that’s Level 2); nor is it intended to examine the learner’s ability to transfer the skills or knowledge from the classroom to the workplace (Level 3); nor does it attempt to judge the ultimate impact of the learning on the business (Level 4). Level 1 of Kirkpatrick’s now somewhat dated “four levels” is intended simply to gauge learner satisfaction.

Typically, we measure Level 1 with a smile sheet, a dozen Lickert-scaled questions about various aspects of the experience. At the end of the list we’ll put a catch-all question, inviting any other comments. I won’t repeat the reasons why the end-of-course environment in which such questions are answered is not conducive to clear, reasoned responses. But the very design of such questionnaires is ‘leading’ and produces data of questionable validity, even in a calm and unhurried environment.

Far too many of the smile sheets that I see put words or ideas into the mouths of learners. We prompt for feedback on the instructor's style, on the facilities and food, on the clarity of slides. The net effect is to suggest to respondents (and to those interpreting the responses) that these things are all equally important, and that nothing outside of the things asked about has much relevance. By not prompting respondents you are likely to get to those things that, for them, are the real burning issues. Open questions are not as simple to tabulate, but they give you an awful lot to chew on.

Now the one-question approach does not necessarily give you all the data that you need to continuously fine-tune your training experience – but neither does the typical smile sheet. Trainers need to understand that sound analytical evaluations often require multi-stage studies. Your end-of-course feedback may indicate a problem area, but will not tell you specifically what the problem is. A follow-on survey, by questionnaire, by informal conversation, or by my preferred means of a brief focus group, will tell you a great deal more than you could possibly find out under end-of-course conditions.

The typical smile sheet is a lazy and ineffective approach to evaluating learner satisfaction. It may give you a warm and comfortable feeling about your course or your performance as a trainer, or it may raise a few alarm flags. But the data that it produces is not always actionable, is rarely valid, and often misses the important issues.

In market research, or any statistical field for that matter, there are two important errors that good research tries to mitigate. Known as Type One and Type Two Errors, they measure the likelihood of seeing something that is not there and the likelihood of missing something important that is there. I have never heard anyone address these error types in their interpretation of Level 1 results.

We see in our smile-sheet results what we want to see, and react to those things that we regard as relevant. If we are so smug in our knowledge that we know what is going on anyway, why do we bother with token smile sheets at all?


Original in TrainingZONE Parkin Space column of 5 August 2005

Meaningful metrics beyond ROI

There is a common misconception in business that, because they work with them all the time, financial people understand numbers. They like to reduce everything to money – what did it cost or what did it make? They insist on dealing in certainties and absolutes, where every column balances to the penny. But the real world does not work like that. The real world is characterized by imperfections, probabilities, and approximations. It runs on inference, deduction, and implication, not on absolute irrefutable hard-wiring. Yet we are constantly asked to measure and report on this fuzzy multi-dimensional world we live in as if it were a cartoon or comic book, reducing all of its complexity and ambiguity to hard financial “data.”

We struggle for hours (often for days or weeks) to come up with the recipe for “learning ROI.” The formula itself is simple, but the machinations by which we adjust and tweak the data that go into that formula are anything but simple. Putting a monetary value on training’s impact on business is fraught with estimation, negotiation, and assumption – and putting a monetary value on the cost of learning is often even less precise.

Yet when was the last time you saw an ROI figure presented as anything other than an unqualified absolute? If you tried for statistical accuracy and said something like, “this project will produce 90% of the desired ROI, 95% of the time with a 4% error margin,” you’d be thrown out of the boardroom. You simply can’t use real statistics on an accountant, because the average bean-counter can’t tell a Kolmogorov-Smirnov from an Absolut-on-the-rocks. Don’t tell us the truth; just give us numbers that conform to our unrealistic way of measuring the business.

We spend way too much time trying to placate financial people by contorting our world to fit their frame of reference, and we allow them to judge and often condemn our endeavors according to criteria that are irrelevant or inappropriate. Perhaps there is some comfort in knowing that the problem is not unique to training. In a couple of decades in marketing, I have seen plenty of good brands ruined by ill-conceived financial policies, usually to the long-term detriment of the company as a whole.

But you don’t need to be a statistician or an accountant to make a strong business case based on logic and deduction, and there is no need to be pressured into using the preferred descriptive framework of a book-keeper. The pursuit of the measurement of ROI in training is often a red herring that distracts from the qualitative impacts that our work has on the performance of the business. ROI is typically not the best measure of that, and, after making all of the heroic assumptions and allocations needed to arrive at it, that magic ROI figure may well be a false indicator of impact.

Unfortunately, the indicators that are useful and reasonably accurate are often hard to convert to financial data, so they do not get taken seriously. And, compounding the problem, training managers themselves often ignore these indicators because they are not captured at the course level. Our focus too often is on the quality of courses rather than on the quality of our contribution to the business in total.

We need to widen the focus. While learner satisfaction, test results, and average cost of butts-on-seats are useful metrics, it is only after our learners have returned to work that we can begin to see how effective the learning experience really was. What are some of the indicators that let us know how we are doing? Many of them are produced already, often by the financial people themselves, and tracking them over time gives good insights into where we are doing well and where we might need to pay more attention.

Some of those metrics include:

• Training costs per employee.
• Enrolment rates and attendance rates.
• Delivery modes, plans against actuals.
• Percentage of target group that is “compliant”.
• Time from eligibility to compliance, or to proficiency.
• Percentage of workforce trained in particular skill areas.
• Learning time as percentage of job tenure.
• Availability, penetration, and usage rates of help systems.
• Skill gap analyses tracked over time.
• Productivity (such as, for example, number of new clients per 100 pitches).
• Attrition rates.
  

There are many, many more. Metrics such as these let us put on the manager’s dashboard indicators of performance in areas such as operational performance, compliance, efficiency, effectiveness, and workforce proficiency, as well as harder to capture dimensions such as motivation and readiness for change. Training departments need to think “outside the course” and come up with ways to derive the right indicators in a way that is inexpensive and unobtrusive.

One of my frequent recommendations is that training departments learn something about data collection from their marketing colleagues, and set up the ability to run surveys and focus groups, to investigate learner satisfaction, customer attitudes, job impacts, attitudes, and manager perceptions. This skill is often absent in training departments, which is a pity because these methods can produce great insights and save money and time. If you build this capacity into your training organization, getting a read on Levels 3 and 4 can become as much a part of your evaluation regimen as gathering smile sheets. You don’t have to interrogate the universe if you can pick a small sample. And you can produce real data and real trends that go down very well in the boardroom.

Original in TrainingZONE Parkin Space column of 22 July 2005

The Learning Object Paradox

Parkin's Learning Object Paradox (PLOP) states that: "The more reusable a learning object becomes, the less useable it is." This is because the usability of a learning object varies in direct proportion to its size while its reusability varies in indirect proportion to its size.

Think in terms of bricks, rooms, and houses. Bricks can be interchanged without affecting the harmony of a house design. Rooms cannot. The smaller your learning objects become, the easier it is to slip them in to other uses without creating any major disruption, but the less “meaningful” they are. The larger the objects become, the less re-usable they get, because they become more context-rich. But you get to a point where the size of the object is large enough to be self-fulfilling and truly meaningful, usually at the level of a house, or whole course.

Which is why, in practice, most learning objects are no smaller than a course. A course is not very re-usable, though you may fit it into different curricula, in the same way that universities fit different courses into different degree programs.

You don’t hear much about learning objects and sharable content objects any more, at least not in mainstream training circles. But there is still an enormous effort going on among learning technologists to make this idea more workable. Are these efforts rather like trying to build a better steam engine long after the internal combustion engine has gained popularity?

The original concept of sharable content objects (in a training context) assumed that learning is primarily content-driven, and that learning content can be decomposed into smaller and smaller components that still retain some inherent independent pedagogical value. If your objects are designed to “click” together, like a child’s construction toy, you can create lots of different learning experiences by clicking together the appropriate objects from your repository of already-created content. The argument went that while the initial cost of building a decomposable course might be higher than building a stand-alone course, the ultimate savings derived from reusing content would be significant.

The historical limitations of learning objects (particularly as manifested in early SCORM) have resulted in dire canned e-learning course design, or have led instructional designers to simply wrap a SCORM interface around whole courses. Today, the “object” is rarely smaller than an entire course. Effectively, interoperability of courses – the ability to run a course on any conformant LMS – has been the primary benefit of SCORM, rather than the reusability of smaller content objects.

The notion of reusable learning objects is in many ways counter to the web ideal of dynamic personalized content, the ISD ideal of internally consistent look, feel and sound to learning flows, and the pedagogical ideal of performance-objective driven custom-built learning experiences.

Re-usability implies that a learning object developed for use in one context can be simply plugged into another context. It also suggests that such an object ages slowly, staying relevant long enough to be reused often enough to justify its reusability. This is more true in some fields than in others. So, for example, you develop an object (say a lesson describing Maslow’s Theory) as part of a management course on motivation. Later, you develop a course on applied psychology and you don’t need to re-create the Maslow lesson because you can simply lift it from the earlier course and drop it into the later course. As the theory itself never changes, the object can be expected to have a long life -- so long as its content is devoid of context, which is typically pretty mercurial.

We have all taken short cuts in the classroom, pulling slides from one session and using them in another, even if they have different layouts or color schemes. That may work better in a classroom setting than online, because there is continuity in the primary medium: the trainer. But online, without the mediation of a live instructor, not only can you end up with all sorts of disconnects in terms of media (different voice, graphic style, look, fonts, pace, interactivity), you are also likely to get disconnects in terms of pedagogical approach. This is avoidable, but only by imposing content standards that are dull and uniform, so every course ends up looking, sounding, and running like every other.

Perhaps the most important obstacle to the success of sharable content objects is the fact that learning is not primarily about content, or about courses. Those who glibly pronounce that “content is king” really irritate me, because it is patently untrue. While content is obviously essential, context and process are more important to learning. But that’s a rant for another day.

Despite all of the reservations and difficulties, the idea of reusability has enduring appeal. But today, as collaborative community-based learning starts to take shape, perhaps we should be re-thinking what a learning object might be. Instead of looking at a learning object as a chunk of easily-connectable content residing on a server, maybe we should be looking at an easily-connectable person residing on a network...


Original in TrainingZONE Parkin Space column of 8 July 2005

Learning evaluation strategies take the mess out of measurement

Why do we spend so much money, and a substantial amount of time, on training? What is the point? Do we just do it because we have always done it, or because everyone else is doing it? Is it because we just like our people to be smarter? Or are we expecting that training will somehow help our company perform better? If it is the latter, how do we know what impact we are having?

I always find it a little disturbing when I come across yet another major corporation that does not have a defined learning evaluation strategy, or that has a strategy which everyone ignores. In fact, I’d say that nine out of 10 companies that I have worked with do not take learning evaluation seriously enough to have formalized policies and procedures in place at a strategic level.

A learning evaluation strategy sets out what the high-level goals of evaluation are, and defines the approaches that a corporation will take to make sure that those goals are attained. Without an evaluation strategy, measuring the impact and effectiveness of training becomes something decided in isolation on an ad-hoc, course-by-course basis.

Without an overall strategy to conform to, instructional designers may decide how and when to measure impact and what the nature of those measures will be, and will use definitions and methodologies that vary from course to course and curriculum to curriculum. When you try to aggregate that evaluation data to get a decent picture of the impact of your overall learning activity, you find that you are adding apples to oranges.

Most companies have done a fairly good job of standardizing Level One evaluations of learner attitudes (smile sheets). Most also collect basic data that let them track activity such as numbers of learners or course days. But these are merely activity-based measures, not performance-based measures. They tell you little about the quality or impact of the training being provided.

Once you start to look at Level Two and up, each course tends to run its own evaluation procedures. In the absence of strategic guidelines or policies, those evaluation procedures can be token, invalid, meaningless, or inadequate – if they exist at all. Even a good instructional designer with a good grasp of evaluation practice may structure measurement measures that, while superb within the context of the specific course, are impossible to integrate into a broader evaluation picture. And the more individual courses require post-training impact measurement, the more irritating it becomes for learners and their managers.

There are many approaches to measuring the impact of a company’s investment in learning that go beyond course-level evaluation. In fact, for the bigger issues, the individual course or individual learner are the least efficient points of measurement. You may decide, for example, that surveying or observing a sample of learners is more efficient than trying to monitor them all; you may decide that a few focus groups give you more actionable feedback than individual tests or questionnaires; you may choose to survey customer attitudes to measure the impact of customer service training, rather than asking supervisors their opinions; or you may opt to select a few quantifiable data points such as sales, number of complaints, production output per person, or staff turnover as key indicators of training success. Your strategy would set out, in broad-brush terms, which of these approaches would be used.

A learning evaluation strategy is not enough, of course. You have to make sure that all of those involved in training design, implementation, and analysis understand the strategy and are able to implement it in their day-to-day work. I have found that the best tool you can give people is a company-specific “learning evaluation bible” that not only lays out the bigger picture, but also provides common definitions, norms, standards, baselines, and guidelines for developing and applying measurement instruments, and for interpreting the resulting data. (I used to call this a Learning Evaluation Guide, but the acronym was an open invitation for way too many jokes).

This document should be a practical guide, rich in examples and templates, that makes it easy for everyone to conform, at a course, curriculum, or community level. The last thing the bible should be is a four-binder bureaucratic manual that looks like it was produced by an EU subcommittee. Rather it should be more like a set of practical job aids, or the guides that are published to help learner drivers prepare for their test.

Without an evaluation strategy, we are left floundering every time someone asks what our return on investment (ROI) on training is. I agree that calculating ROI is problematic, especially at the individual course level, and is often unnecessary. But if you are spending millions on completely revamping your sales-force training curriculum, you’d better be thinking about how to measure ROI, and build those measurement requirements in up front. You would not invest hundreds of thousands in an LMS unless you were convinced that the investment would bring an acceptable return, and you naturally will want to measure how well the investment performs against plan.

At a departmental level, an evaluation strategy helps us to answer that awkward ROI question, not with defensive rationalizations, but with coherent, consistent evidence that supports the contention that our training investment is indeed achieving its desired results.


Original in TrainingZONE Parkin Space column of 8 July 2005

Privacy is every trainer's business

Some years ago I spoke on the topic of managing privacy in e-learning at a large learning conference. Only three people showed up. Two of them were expecting a session on how to keep distractions away from employees trying to e-learn in a busy office environment. I’m not sure that the awareness or concern of trainers has been raised much since then, but it needs to be.

The recent admission that 40 million customers of all the major credit card companies have had their data hacked is the latest in a mounting wave of failures to secure customer privacy. In the past few months alone, the confidential information of tens of millions of people has been “let go” by household-name companies in banking, finance, insurance, education, and retailing.

The criminals get the bad press. But these are corporate outrages, because in all cases they could have been avoided had the companies to whom customers entrust their data not been so inept, or worse indifferent, about their responsibilities. And they continue to get away with it because their customers themselves are ignorant or indifferent. Or, in the case of credit and debit cards, customers have no choice – if all card companies are equally bad, and all continue to have the same data intermediaries in common, you either accept the risks or try to live plastic-free.

What does this have to do with training?

First, those who are providing training in any subject, either in-house or as a vendor, online or in class, need to review their own procedures and policies for securing the personal data of their learners.

Whether they are paying customers or not, with the help of learning management systems we are gathering more and more intimate details about each learner. Those details need to be secured, not just from outside hackers but from any internal management and training staff who do not explicitly have a right to access. And learners need to know that they are secure.

You have a significant moral and motivational obligation to guard the privacy of your learners. Once you post your privacy statement, you are legally bound to adhere to it. When learners think that potentially every keystroke, decision, response, and test result can be tracked, they get nervous. When they think that their managers have access to those details, they worry. And if they think that their data may become publicly available, they may rebel.

There are many simple things you can do to make learners more comfortable. Among them:

• Post prominently a well-formulated privacy policy. For each course, restate this policy and provide any specific elaborations or differences for that course. Have each learner accept that policy as part of course registration.
• Tell them what information is collected and what it is used for. Tell them who can access it and under what circumstances. Tell them how it is secured both in databases and in transit.
• If you change your privacy policy, let every current and past learner know about the change and its implications.
• In online courses, let learners pick their own user ID and password, rather than automatically allocating them their name or e-mail address. Unless it contravenes a learning objective, give them an option in chat rooms or threaded discussions to use anonymity.
• If you use cookies, use one-time self-terminating session cookies so you are not placing track-able cookies on their PC.
• Overtly make use of encryption where learners have to provide any personal information.
• Destroy learner data as soon as it is no longer needed. Archive needed learner data offline or securely behind firewalls. Lock up your back-up tapes.
• And let learners view their own learner record at any time, so they can see what data is actually available to authorized parties.

The second aspect to privacy is this: there is an urgent need for training in security and privacy for all personnel in any business that accepts and uses customer information. That’s just about every business. (In my view, such training should be government mandated, but then I’m a vendor).

Most security breaches are not the result of hi-tech brilliance on the part of the thieves, but of human weakness on the part of company employees. All employees need to understand the risks and learn the operating habits that mitigate them. They need to appreciate that technology is not in itself an adequate protection, and must be trained to develop the “street smarts” that will help them avoid the common behavior pitfalls so often exploited by villains.

Managers need to get their heads around the policies and procedures that will protect their customers, and must regard these with as much earnestness as those that protect their company.

Privacy and data security are not an IT-only responsibility, nor are they issues that you can deal with after the fact. Training has an important role to play. Get it right at the planning stage and you will be fine. Get it wrong, and you could be in big trouble.


Original in TrainingZONE Parkin Space column of 24 June 2005

Learning innovations

I have just spent a couple of days at a small highly-focused symposium titled “Innovations in E-learning.” It was put together by the US Naval Education and Training Command and the Defence Acquisition University (DAU), who have among the best and brightest training minds that the American taxpayer’s money can buy. They are not short of budget, manpower, or technology, and they get to mess with lots of experimental stuff. I decided to participate because the future of learning matters to me, and because a couple of my virtual colleagues were pretty much dominating the presentations in one stream.

For several years now, Training Departments have been transfixed by the evolving internet in the same way that dinosaurs were probably awe-struck by the approaching comet. So what does the future hold? I’m happy to report that learning will thrive, but trainers will have to merge back into operational roles. Oh, and Training Departments are dead, at least as we know them. As are Learning Management Systems and any other relics of centralized distribution of learning. Learning that is informal, collaborative, contextual, real-time, and peer-generated, will be the mode of tomorrow.

It seems counter-intuitive that military types whose culture is defined by command and control hierarchies would advocate devolution of learning to the swab on the deck-plates or the grunt in the foxhole, but that was the gist of what was being said. Admittedly, it was not being said by the JAG look-alikes or their entourages, but by the civilian gurus who write their white papers for them. And devolution of learning does not necessarily mean relinquishing control – in fact there are some very scary big-brother systems being deployed that (allegedly) will tell anyone with access pretty much what any individual sailor anywhere in the world had for breakfast last Tuesday and, to five decimal places, what his or her competency rating is on any given skill. It is hard to reconcile what they are saying with what they are doing, until you realize that, because these systems are so vast, they take a long time to build and deploy. So at any point in time the military are rolling out systems and policies that have long since been abandoned for something new – which may not see the light of day for a decade.

I was mainly interested in hearing what folks like Jay Cross, Clark Aldrich, Harvey Singh and Ben Watson had to say about workflow learning, collaboration, and simulations. However, in amongst their sessions was a real eye-opener from a VP at IBM. IBM used to be a blue-suit red-tie operation as monolithic as a bank, but it has been doing a lot of shape-shifting in recent years. These days any organization that is unwilling or unable to do that is unlikely to be around very long. It’s Darwinian – those who can adapt most readily are most likely to survive in times of rapid change. IBM’s consulting wing, adrenalised a couple of years ago by their acquisition of Price-Waterhouse Coopers consulting, is doing what big consulting firms rarely do – they are advocating unique solutions that they don’t already have parked in a truck around the corner.

Here’s a quick version of the IBM line on “embedded” or workflow learning:

The most profound shift that will take place in training over the next three years is a movement away from traditional, formal, course-based learning (classroom or online) and towards clever integration into the workflow of learning-enabling tools like Instant Messaging and informal collaboration processes. As we move learning from its “separate service” role to a more integrated coal-face role, one of the biggest obstacles is the political question of who owns it. The other is the need for a deeply rooted culture of collaboration throughout the organization.

A simple example of workflow learning in action: Tom in Finance gets an urgent request to authorize foreign travel funds for an executive. He learned how to do that in a training course last year, but has never needed to do it in practice, so he’s lost. The help system, typically, doesn’t. The FAQ gives no guidance either. So he sends out a broadcast Instant Message to a small group of SMEs and experienced practitioners asking for help. So far this is not a lot different from “prairie dogging” – popping your head up above your cube divider and yelling “Does anyone know how to…” But here is where it gets interesting. Jill, an experienced practitioner in another city, responds to the message. She remotely takes control of Tom’s computer and talks to him as he watches her go through the steps on his screen. She identifies that the help system, the FAQ, and possibly the original training are inadequate, and updates the FAQ in wiki-like fashion. Then she identifies a group of Tom’s peers who might benefit from knowing what Tom now knows, and sends them an announcement of a ten-minute webinar for later that week. During the webinar, she records the session, and saves it to the system where those who could not make it, or those who may encounter the problem in the future, can easily find and watch it. Then she notifies those responsible for basic training, and those responsible for the help system, that they might need to pay attention to the issue. Tom, in the meantime, evaluates the help he has received, and his ratings and comments get added to Jill’s profile for reference by future aid-seekers, and her management.

The technology is not complex, or even expensive. Most people have it on their computers already. Aspects of this are widely used already in e-commerce and e-customer support. Individuals already learn this way intuitively. What is hard is achieving the mindset and the culture that allows and encourages this to happen across an organization.

There is nothing revolutionary in the IBM vision. If you have followed those who advocate informal learning and collaborative learning (and indeed many of my own rants), you will realize that the ideas are not new. But, for me, the amazing thing was to hear them coming from IBM. If Big Blue is advocating this approach, and is actively setting about trying to get it to work in its clients’ cultures as well as its own, then there is something serious going on. Workflow learning has moved from the drawing board to the boardroom. They say that in theory there is no difference between theory and practice, but in practice there is. IBM is taking its theories on the road, and, in practice, is being taken seriously.


Original in TrainingZONE Parkin Space column of 10 June 2005

Knowledge managing the retirement brain drain

In most Western countries, the baby boomer bubble is causing concern for those planning pension and social security services. It should also be worrying employers. A quarter of current American employees will be retiring within the next five years. If the outgoing masses know anything of value and it is not being passed down to others in their companies, those organisations face a brain drain that could harm their ability to operate. While much of the work done in examining this issue is carried out by those seeking to market a solution of some kind, the workplace realities, on which they all seem to agree, are fascinating. It appears that finance departments have little regard for the enduring value of an employee’s lifetime of work.

Accenture recently surveyed several hundred employees who qualified as “approaching retirement” to find out what, if anything, their employers were going to do about retaining their knowledge before they left. (I personally found the sample demographics a little alarming: since when are people between the ages of 40 and 50 approaching retirement, unless of course they are in US government jobs?)

It seems that companies are doing very little to capture that knowledge. Could it be that companies simply don’t value what their “ageing” employees contribute? There’s plenty of anecdotal evidence to support that view. If faced with recruiting or promoting either an inexperienced 25-year-old MBA or a 50-year-old veteran, the MBA is always the odds-on favourite – even in a country where overt ageism is illegal.

What opportunities are offered to imminently retiring employees to pass on their wisdom? According to Accenture, one company in four makes no effort whatsoever to capture the workplace knowledge of retirees, and a further 16% of companies expect retirees to have an informal chat with colleagues before leaving. That’s more than 40% of companies that have no formal processes for retaining expertise.

When you think of the money and time that has been put into training and developing the expertise that is apparently seen as disposable, you have to wonder how serious companies are about the value they place on “human capital”. After all, what is human capital if not expertise? If it is disposable, it has no significant value. And if the expertise of your most experienced people has no significant value, why on earth are you wasting your time bandying about training ROI calculations? At the end of the day, the return on all that investment, in Kirkpatrick Level Four terms, is assumed to be not worth the trouble of securing. Or so your accountants will tell you.

Now, you can look at the other side of the coin and say that more than half of all companies do make an effort to hang onto that expertise. In fact 20% of companies claim to put their retirees through a knowledge transfer process that lasts several months. But I suspect that this is only in exceptional cases for particularly high-ranking employees.

It may be that the failure to make an effort to preserve workplace knowledge is not because such knowledge is undervalued but because the extent of the problem is not realised. In that case, how do we get senior management to sit up and take notice? And, once they do appreciate the scale of the impending problem, what can be done to move awareness to effective action? Perhaps this responsibility falls more squarely on the shoulders of HR management and human capital strategists, rather than on trainers. Or perhaps it is the responsibility of those charged with knowledge management.

Knowledge management (KM) has had a bit of a rough ride over the past 10 years, having made many of the same mistakes that e-learning made. Their focus was initially on building technology-based tools to extract, retain, and retrieve what was in the heads of employees, and their focus was on explicit knowledge rather than tacit knowledge. Nowadays tacit knowledge is recognised as being more relevant to performance, even if it is harder to capture. KM people are working more with informal approaches such as story-telling to capturing what this less tangibly expressed expertise.

The problem is that the less formal our processes become – in both training and knowledge capture – the less easy they are to sell to the corporate bean-counters. And the less tangible our activities are to those who like dealing with hard numbers, the less value they are assumed to have.

I sometimes think that accountants are the biggest obstacles to progress in corporate learning and knowledge management. They love structure and hierarchy and abhor ambiguity and fuzziness. If they can’t measure it, it has no value. How 20th century! We have all run more than our share of the mandatory “Finance for non-financial managers” courses. Perhaps it is time to lobby for some mandatory “knowledge for non-knowledge managers” courses…

A trainer's output is performance, not courses

In Kirkpatrick terms, most “learning professionals” don’t like looking beyond level two. Because it is generally considered too difficult to evaluate training once learners leave the classroom, they have a good excuse for setting themselves much more easily achieved objectives.

A couple of weeks ago, in a reputable training forum, I heard a trainer proudly declaring that his learning objectives were always cast in such a way that they would be fully achieved by the end of a course. I was appalled, and said so. My reservations about competency-based learning management aside, surely instructional designers and trainers seek to impact the performance of learners once they return to the real world? If that is true, instructional designers and trainers have a responsibility for the ultimate on-the-job effectiveness of their handiwork. And, if they have a responsibility, they should set objectives to know whether or not they are meeting that responsibility.

I was stunned by the number of people who came back at me as if I were mad, declaring adamantly (and somewhat self-righteously) that their responsibility for learner performance stopped at the door of the classroom or at the closing of the e-learning browser window. What happened after training was not their problem. If instructional designers as a group have such a limited perception of their importance to the business, then whoever is training them is doing a lousy job.

The question "what things will participants be able to do at the end of the session that they were not able to do pre-session" is a common cop-out that does not produce learning objectives. Rather, it produces a reverse-engineered set of often grotesquely constrained learning promises that have as much substance as those of a used-car salesman.

I am increasingly wary of any learning objective whose time horizon is only the end of a course. By the end of any learning experience, learners may have acquired some new competence. But within months that competence should have grown, been applied, and be achieving business results. That scope should characterise our learning objectives. To be satisfied with taking responsibility only for that which is directly within our domain, important as it is, ignores the fact that learning has a greater business purpose.

Setting objectives that do not look beyond the point where a learner hands in his/her smile-sheet is abdicating responsibility for the effectiveness of training. If course-end is the training performance horizon, then post-course drop-offs in recall or ability become acceptable to trainers as "not my problem."

You could argue that as long as the trainer has created an acceptable change between pre-training ability and post-training ability, then he or she has performed successfully.

But learning is not only about acquiring temporary knowledge and skills. Motivation is an important part of what a trainer should be doing. At the very least, we should design our courses and conduct them in such a way as to motivate people to want to learn, to want to keep learning, and to want to apply what they are learning once they leave the training environment. If that motivation is absent, we have not done our job.

Admittedly, there are many factors beyond the control of individual trainers that impact transferability of learning to the workplace. Management and systemic obstacles all affect the way people grow in their jobs. But trainers can’t simply ignore these. We should try to exert influence even if we have no overt responsibility. And if those obstacles are immovable, we should design our training to accommodate them and to equip learners to deal with them.

Simply saying "not my job" is an inadequate response in someone who has taken on the task of improving the performance of fellow employees. Trainers and instructional designers do not have total responsibility for a learner’s performance on the job, but I think they should share some of that responsibility. If their salaries, bonuses, or careers depended on it, we’d see much more effective learning experiences being developed.

E-learning salaries

Recently the E-learning Guild released the results of its annual e-learning salary survey. Data was gathered in the first quarter of this year from people who work in the e-learning field.

It’s amazing that they were able to find any participants for the survey, since, in the US at least, you hardly ever see a job advertised as an “e-learning” position any more. Two or three years ago, they were all over the place, but now it seems that some facility with e-learning is simply one of the requirements of anyone looking for employment in the broader field of corporate learning.

So while the results may be a little questionable, they do offer an interesting insight into the relative values placed on different skill-sets and responsibilities by companies in America. For those who responded, what do US pay packages look like?

Across the country, the median annual pay of a Training Manager working in e-learning was $74,000 (£40,200 for those of you in the UK).

The US is a big place, and salaries vary with geography, jobs in the middle of the country normally paying 10-20% less than the East or West coast. Washington DC, where I am, is actually part of the South East, and salaries are typically a little less than those paid in New York, just to the north. So while New York Training Managers pulled in $84,000 (£45,600), those in DC settled for around $72,000 (£39,100).

Higher up the ladder, the national median salary of a Training Executive was $120,000 (£65,200). At the bottom of the scale, a classroom or online trainer typically earned $50,000 (£27,200). Between the humble trainer and the training manager, fell all of the “specialized” folk, including curriculum designers and developers, instructional designers, and content developers.

How hard do people have to work to earn these salaries? Permanent full-time employees are averaging 49.9 hours per week, and they get, on average, 18.8 days of paid vacation each year. (One poor soul reported working 80 hours a week! Been there, done that, dodged the bypass.)

The question that jumps out at me is this: why are trainers, who contribute so much to the actual effectiveness of training, so obviously undervalued by their organisations? Is an instructional designer really worth 20% more than a trainer? Trainers, after all, have the ultimate responsibility to “make it work” even if the design is no good. Given the insipid nature of instructional design that I see so often, I have to believe there is something wrong with this picture. Perhaps it is simply supply and demand. Or perhaps it is that, in companies that have gone overboard with e-learning, the classroom or synchronous trainer has indeed been sidelined.

You can get the complete report, free of charge, at the E-learning Guild site, but you have to sign up, free, as an associate member.

Training Needs Analysis needs analysis

I have yet to meet anyone in the learning business who will question the value of a Training Needs Analysis (TNA). Indeed, there are many who will insist that a TNA is carried out before any work is done on defining or developing training of any kind.

We all know, understand, and value the concept of a TNA, and believe that, in a perfect world, we’d invest time and resources in TNAs whenever we could.

That’s why I was initially surprised when a debate erupted recently over the definition of Training Needs Analyses. On reflection, though, it is clear that “TNA,” rather like “e-learning” or “evaluation” is one of those Alice in Wonderland terms that means exactly what you want it to mean.

As a result, half a dozen learning experts can carry on a conversation about TNAs all day, not realising that they are each talking about rather different things. And a training department can commission a consultant to carry out a TNA, and neither party will know until it is too late that they were each working with very different concepts.

To some, a TNA is a big-picture strategic process that helps define which performance gaps are best addressed by training, and which are best addressed by other means. To others, it is a project-specific tactical analysis which assumes that training is called for and seeks to examine the learning environment and define the optimal instructional processes. To others still, a TNA falls somewhere in the middle, or is an operational process designed to aid in planning and budgeting.

But whatever the motive or focus level, most surely agree that the outcome of a TNA is a clear idea of what has to be done, how it should best be approached. The TNA may also tell you what the recommended approach might cost and what return might be expected.

My own perception of Training Needs Analysis is as a bridge between the strategic and the tactical. I see a TNA as being most useful at a project level, where a specific performance gap has been identified, and objectives to close the gap have been defined.

A TNA would tell you how much of that gap might be addressed by training and at what cost. And, if training is subsequently allocated a role, a more detailed TNA would define precisely what learning objectives would be met, and in what way.

Unfortunately, reality often intervenes. The time and cost taken up by a formal TNA is frequently just not worth the potential benefit. The individual judgment, instincts, and insights of those in training management can often produce a much faster, more practical ‘analysis’ than a formal study. And, in truth, the only time it is worth doing something more formal is when the cost of making a mistake is simply unacceptable.

There is absolutely nothing wrong with this approach. It is a perfectly valid way to make decisions. And, despite its not being a formal research-based study, it is still a TNA. To me, a Training Needs Analysis is any analytical process that seeks out and examines relevant information, qualitative or quantitative, with the objective of determining the nature of the role that training can or should play in resolving a performance problem or exploiting a performance opportunity.

In the real world of market research, multi-million dollar budgets are frequently decided on the mutterings of a few dozen people in focus groups and the collective experience of the marketing decision-makers – simply because a statistically significant “proper” research project might take too long, cost too much, and produce a result that you can’t take to the bank anyway.

I am, sadly, an old-school geek, having spent a large part of my academic and work life in the market research field. As a result, I get unreasonably excited about anything that pretends to be a rigorous analytical study but is, in reality, a sloppy or token gesture. I am not talking about the calculated expertise-based decision-making approach mentioned above, but about the attempt to pass off Pseudo-studies as proper research and analysis. I know how easy it is to “prove” just about anything with research, and how assuming what you want to prove can result in a study that conveniently reaffirms your expectations.

I have seen many “TNAs” do just that. They are designed to discover a set of needs that magically coincide with a pre-determined solution. Company training departments are less guilty of this than large outsourced training vendors, who have a rather fixed set of offerings to sell.

They find it more expedient to “customize” their customers’ perceptions of their needs, than to customize the nature of their solution. Probably the worst culprits are the big consulting firms who have a batch of “processes” in their database that they try to surreptitiously syndicate across as many client companies as possible, all under the guise of original tailor-made work. They use token TNAs to do the selling job for them, and usually get paid substantially for those TNAs as well.

It has been said that it doesn’t matter what you call it so long as you do it well. I disagree. While we may never reach an industry-wide consensus, aligning our thinking on what a TNA is does matter. The more we outsource the different aspects of the training role, the more open we become to confusion, ambiguity, and waste, especially if we don’t take the time to define our terms and – more importantly – demand that our business partners define theirs.

Learning from Google

I suspect that, should humanity survive the next few hundred years, we’ll look back at the early decades of 21st Century and think of that time as the learning revolution, much as we now think of the turning decades of the 19th century as the industrial revolution.

We’ll see the sacred cows of educational dogma being put out to pasture, and the previously ingrained best practices of adult learning sneaking out of town under cover of darkness. And while university professors will probably be with us forever, training departments may be decimated and their inhabitants diasporised into all the far-flung operational units of organisations.

Like most significant revolutions, it takes time. And those most affected by it are unaware of the significance of what is happening. The harnessing of fire, the invention of the wheel, the concept of zero, the discovery of penicillin, all had irreversible impacts on the nature and pace of change in the societies that reluctantly embraced them.

The evolution of learning has been stunted for many centuries, simply because until recently the volume of knowledge did not grow significantly, the pace of change was pedestrian, and the nature of change was linear and predictable. Our approach to learning was adequate for those circumstances. Not any more.

A “Renaissance man” of the 1600s may have been more comfortable with languages and the classical arts than your average high school student today, but had a great deal less knowledge. Even 60 years ago it was possible to put all of the significant knowledge in all fields of endeavour into a few bookshelves. In 1943 the Chairman of IBM estimated the total world demand for computers at five. Today the world generates more new data in a day than our grandparents were exposed to in a lifetime. Five years from now, the information base of the world is expectedto double every eleven hours. You can no longer cram everything there is to know about a particular subject, let alone all the knowledge of the world, into a single head. And if you could, you’d never be able to find what you needed in time to be effective.

Small wonder that, at $71 billion, Google now has the world’s largest market capitalisation, just ahead of Yahoo, while industrial-age giants like Ford and General Motors have been downgraded to junk bond status. We now place a lot more value on those who can instantly find information than we do on those who merely manufacture “stuff”.

You won’t see a training company in the Fortune 100. Neither education nor training has ever been an investors’ industry. In looking for funding for e-learning and online simulations businesses over the past 15 years, I have done the rounds of venture capitalists in Europe and the US. More times than I care to remember I have eaten club sandwiches on the terrace of Sand Hill Road’s Sundeck restaurant, pitching ideas to the luminaries of the Silicon Valley VC community, and the general perception of training is that there is no exciting return to be made. A VC once told me that, to investors, the benefit of corporate training is a mirage – everyone sees it differently, everyone wants to believe in it, but when you try to pin it down it vanishes. Another asserted that, if improving corporate performance is really important, there are usually more productive solutions than training.

Training, as we have known it, is not driven by consumer demand, has uninteresting ROI, provides really fuzzy benefits, and cannot be easily differentiated. But that can change. Education focuses on helping people know more; training focuses on helping people do more; Google focuses on helping people find out more – when they most need it.

Corporate learning has to follow the Google “search & connect” model instead of the General Motors “produce & sell” model. Training purists sneer at “just-in-time” help systems, insisting that people need to know how to do things themselves. They undervalue collaborative learning networks, regarding them as somehow cheating. They fervently believe that adult learners must be led, child-like, through pre-determined learning paths mapped out and controlled by a central authority. They gauge the worth of an employee by his or her ability to survive on a corporate desert island, bereft of books, colleagues, mentors, databases, systems, or communication.

Get real! We no longer need to rub two mental sticks together to make intellectual fire. We no longer need to see ourselves as slaves tasked with caring for children. Pedagogy is a framework for understanding mental processes, not a job description. And andragogy is often abused, its “science” used to justify perpetuation of training processes that are simply no longer relevant or appropriate. I’d suggest the concept of “googlegogy” were it not for the fact that Google itself may not be top dog for long.

Twenty years ago the internet triggered a "Big Bang" in knowledge and personal communication. E-businesses have made rapid progress in exploiting the opportunities as they emerge, building real-time e-commerce systems, personalised services, and ubiquitous access to information. Trainers and instructional designers, however, are still building courses. We need to start thinking about new ways to foster learning that accommodate the real needs of learners and leverage all the data, systems, economies, and connectivity that “digital natives” now take for granted.

The needs of learners should be defined in marketing terms – instead of focusing only on learning content, learning objectives, and the outdated and inadequate concepts of learning “styles”, we should regard learners as customers and craft our learning processes to match the processes by which they want to learn. Instructional designers need to adopt some marketing thinking, so they have a better chance of being able to understand learners according to criteria that learners consider important. And they need to see the end product of their endeavours to be delighted customers instead of polished courses.

Maybe then trainers will have a better chance of surviving, or even guiding, the learning revolution.

Competency based learning management

For some years now, the notion of competency-based learning management (CBLM) has been viewed by training departments as the Holy Grail. But, like many things that make a lot of sense in theory, the practice is somehow unconvincing. In fact, the pursuit of CBLM, like the pursuit of e-learning, can cause us to lose sight of what is really important. Worse, it might do real damage to a company’s viability.

Here, in a nutshell, is the argument for CBLM: You can break the strategic plan of a business down into resource requirements, and within the human resources you can identify what competencies are required to fulfill the plan. You can drill down into that plan and create ever more detailed lists of competency requirements, by department, by project, by team, and by individual. Then, by surveying in depth the existing competencies of your employees, you can map out the gaps between what we have and what we need, and define a training plan to fill those gaps. No individual needs to be trained in anything that is not directly relevant to his or her personal competency growth path. You save time and money by surgically targeting all of your learning interventions, and doing away with the training smorgasbords of old. And you know that your training plans are intimately linked to your business objectives.

In a word, efficiency: which is management-speak for saving money. (True efficiency is about optimizing outputs and inputs, not simply seeing how far we can cut costs before the house of cards collapses). CBLM is an industrial-economy concept, where specialization and division of labor are considered to be good things, and where no resources should be wasted on helping any individual to learn anything that is not specific to the performance of his or her particular task.

Of course, all training has always been competency based, only it was typically bottom-up, not top-down. I am not performing well as a salesman, it is deemed that I am less than competent, and sales training is in order. But the bottom-up approach, however pragmatic and situation-responsive it may be, does not collectively amount to a clean, efficient fit with corporate top-down approach. And it makes planning and cost-justification difficult. Hence CBLM. (As an aside, competency-based HR management, where skill gaps are looked at as resource requirements rather than training needs, has contributed to much of the outsourcing pandemic in Western corporations. Why spend time and money training people with the hope of a 10% improvement in performance, when you can fire the lot and hire offshore resources with greater competency at a third of the cost?)

Here, in a nutshell, is my argument against CBLM: We are no longer operating in an industrial age economy, so competency-based learning management doesn’t really work.

And, in the rare instances where it can be made to sort of work, it handicaps the organization’s ability to rapidly innovate and adapt. Development of employees is now as important as their training. Collaborative workflows require employees to have a broader, not narrower, vision of their work. Industrial economy thinking seeks to eliminate overlaps in competencies; knowledge economy thinking acknowledges that without overlap there can be no synergy, and that competency overlaps foster the capacity for continual innovation. It is the opposite of specialization. Today, the more co-workers there are that know what I know, the more valuable that knowledge becomes. The same is true of skill. The future prosperity of organizations emerges from the white spaces between the org-chart boxes – sanitize those spaces by scrubbing regularly with CBLM, and nothing of value will ever grow there. Once you have instilled a regime of rigorous CBLM, don’t ever dare to tell an employee to think outside the box.

Informal learning is largely instigated and managed by individual employees, and the inherent contradiction between fostering informal learning and trying to centralize efficiency-driven competency-based learning management may not be in the best interests of corporate performance.

Another significant flaw in CBLM is the belief that we can adequately measure existing competencies, and can manage the ongoing maintenance of the resulting databases. Apart from the thin-ice of actually defining and measuring each competency in a way that is ethnically/culturally neutral and legally defensible, the effort inherent in keeping records up to date leaves most companies flailing. With e-learning, the real savings are not in course costs but in the business process costs associated with learning. With competency-based learning management, the business process costs may dramatically outweigh the course cost savings.

Is all CBLM a waste of time? Of course not. But blindly pursuing it because that’s what everyone else is doing, or because that’s what all the software vendors are hyping, can lead to the same frustration, expense, and lost opportunities as did the blind pursuit of e-learning. Like e-learning, which ironically has given enterprise-wide CBLM the semblance of being achievable, CBLM is only a good idea where it is appropriate, and where it is interpreted and applied intelligently and pragmatically. And like e-learning, which many companies wrongly feel they can fire up and leave running “as is” for eons, all CBLM initiatives are in a state of perpetual rapid decay, like a snowball rolling down a warm hill. A CBLM system, in a company competing in rapidly-changing dynamic markets, has a half-life of mere months.

Factor that into your efficiency calculations, and weep.

Original in TrainingZONE Parkin Space column of 22 April 2005