Monday, January 31, 2005

The self-delusion of smile-sheets

There is an unhealthy tradition in training that every course must end with participants filling in a questionnaire in which they express their feelings about their learning experience. We ask people scurrying for the door to take time to check boxes on multiple Likert scales and to scribble a few comments. Then we gather up the questionnaires and flip through them hoping desperately for a few feel-good moments of positive feedback, and finding good rationalizations for any negative scores.

Then, if the training organisation is vaguely professional, the questionnaire results get incorporated into a database where they are manipulated to produce trends and averages. Attention gets paid to those aspects that learners are clearly less happy with, and trainers with consistently good average scores acquire a sort of heroic status. Oh dear.

I can’t think of too many other fields of activity where we try to measure customer satisfaction immediately after the event each and every time. Does the waiter refuse to let you leave the restaurant till you have filled in the form? Do you get accosted by questionnaire-wielding cinema managers during the closing titles of a movie? Does your dentist insist that you check a few boxes before you are allowed to escape the chair?

I used to work with a training company that evaluated the competence of its trainers based on their smile-sheet score averaged across all questions and all participants in a particular session. Trainers would come in beaming with the news that they'd just run a 4.7, as if the number actually meant something (though of course it did -- their daily rate was pegged to their perceived ability to make learners happy, not to the impact that they had on the effectiveness of those learners once back on the job).

My background is in market research, so I know a thing or two about questionnaires and survey design. Smile-sheet questionnaires are often really badly designed, with ambiguous questions or counter-intuitive formats. The process is far from ideal, with tired respondents working in haste. And the environment is inherently invalid, with respondent-trainer relationships interfering with objective perceptions or honest responses. The data that comes in is badly flawed, and those flaws are magnified with every manipulation and interpretation. Garbage in, landfill out. Evaluating training quality by smile-sheets and then taking action based on that evaluation is naive and delusional.

Technical validity issues aside, we all know that smile-sheet scores are not a meaningful measure of training effectiveness. Nor do they provide any real insight into what, if anything, a participant has learned, or the impact that it might have on that person’s real-world work performance. They are at best a blurred snapshot of customer satisfaction at the time, nothing more. But we continue to use and revere them.

As trainers, we should have the sensitivity and curiosity to be continually taking the pulse of our trainees and adjusting accordingly. So why do we perpetuate the smile-sheet ritual? Is it because “management” wants some easy means of monitoring trainer performance? Or is it because learners have come to expect some kind of formal feedback process beyond the ability to speak up in class?

Whatever the motivation, we are simply not measuring the things that we are reporting on. There are a series of large conceptual leaps between what is actually measured (vague momentary happiness) and what we interpret as being measured (training effectiveness). The endless discussions that go on about whether a 5-point scale is better than a 7-point scale, whether checking boxes is better than circling numbers, whether semantic differentials (marvelous - - - awful) are better than numeric intervals (5 - - - 1), are all irrelevant if the core concept is inherently invalid.

If we really care about the quality of the learning service we are providing, we should put a little more effort into measuring and monitoring that quality. Quality is all about the extent to which we achieved the learning objectives, and that cannot be measured at the point of training. It has to be measured after the event, where the training is applied on the job.

We should at least care enough to acknowledge that the smile-sheet is only a token gesture at quality control, and devise and implement systems or processes that are more incisive and significant. We pay lip service to Kirkpatrick’s four levels and wistfully wish it were easy to measure beyond level one. But that is rarely part of our brief, so we let it slide. How much easier would it be to do all of those ROI projections if we had at least some valid data to work with beyond the fact that we scored a smile-sheet 4.7 average in the courses we ran last year?


Original in TrainingZONE Parkin Space column of Jan 21 2005

Sunday, January 30, 2005

PowerPoint in perspective

PowerPoint is audience-abusive, presenter-addictive, and information-transfer sclerotic. But with a little determination, it is possible to stop it from calling the shots and take back control of your communication.

There is no end to the supercilious put-downs of the tool that roll of the tongues of those who know better. “Death by PowerPoint” is no longer funny, but we all use it to show how much smarter we as audience members are than those making an effort to communicate their message from the front of the room. Yet the same people who sit in meeting rooms and sigh inwardly as another bullet point whizzes in from stage right, will themselves inflict similar discomfort when called upon to present.

Not all trainers are detractors. There are many PowerPoint apologists who will tell you that it is a perfectly good tool, and that the fault lies with the way people use it. Not true. It is a tool that is often quite inappropriate for the task, yet our norms and expectations dictate that we use it anyway.

PowerPoint leads you to communicate in a slide-show or lecture mode. It imposes a sage on stage relationship between the audience and the presenter, and it makes the content as manifested way too central in the process. Content is of course important, but PowerPoint forces an abstracted sequentialism on content that often makes it incoherent. The presentation typically becomes the communication, instead of merely supporting it. Dressing it up with animated transitions and pointless clip-art will not stop the haemorrhaging of audience attention. The worst offence of PowerPoint is that it often serves as the velvet curtain behind which the Wizard of Oz presenter hides his/her own lack of substance.

If you have to use PowerPoint, what can you do to make the best of it? Assuming that basic legibility issues and presentation skills are already in hand, here are ten tips in, er, bullet point format:
1. Understand your strategic objective and structure your communication so that it gets you there. If you are trying to teach, use appropriate communication processes and don’t let “your presentation” become a substitute for a proper learning experience.

2. Only use PowerPoint slides for those parts where it makes the communication clearer or more efficient. If you can’t communicate your message verbally without a PowerPoint prop, perhaps you have nothing worthwhile to say. Don’t feel compelled to have a slide for everything you are going to say, and use blank slides liberally as spacers that bring focus back to you or to your audience.

3. Don’t use PowerPoint as a pacemaker. If your presentation design interferes with the right pace of communication, rethink it.

4. Design to express ideas, not merely to summarize. A bullet list is often the last refuge of the poorly prepared.

5. The time and attention of your audience is a scarce commodity – respect it. Self-indulgence seems to be encouraged by PowerPoint, and results in tedium and time pressure.

6. Choose the visual media that bring most clarity to any item. Numbers are often more effective than text or graphics, still images can sometimes do more than busy animations. For a real-time electronic whiteboard, plug in a cheap graphics tablet and write on your PowerPoint slides.

7. Don’t be afraid to crowd a slide where appropriate. An individual item often needs the context of other items to be meaningful, and a linear revelation may not be the most effective way to go.

8. Keep it simple and elegant. Your slides should support what and how you are communicating, not dominate it.

9. Stay flexible. For important events such as conferences, I usually have two versions of my presentation (one high-res full-color, one lower-res 16 color) because you never know what limitations the projector may have. If I am presenting as part of a panel, I prepare the full presentation and a stripped-down version in case someone steals all my time. And I make sure I can talk comfortably without any aids at all if necessary.

10. Respect your audience’s need for a record of the presentation, but don’t take that need literally. Even the most text-heavy PowerPoint slideshow is usually an incoherent representation of what you actually had to say. Instead, take the time to write a paper, including any relevant illustrations from your presentation.

In skilled hands PowerPoint can be quite effective even when it forces a sub-optimal mode on the communication. But “presenting” is rarely what we should be doing. Training, selling, persuading, inspiring, or informing all require different communication modes, processes, and media. Why do presenters and audiences have an unquestioning expectation that they will meet their objectives with a slideshow? Sometimes technology becomes an end in itself, and PowerPoint has achieved that status. The medium has become more important than the message.

You are often better off simply turning off the projector and talking with your audience. If you need to illustrate your talk, try using a less linear tool, such as HTML or Flash. But there is not a lot you cannot do, very effectively, with a whiteboard or a flipchart.


Originally in TrainingZONE's Parkin Space column of Jan 14 2005.

Monday, January 17, 2005

An eventful year for training technologies?

In 2004 technology appeared to be coming out of the post-bubble-burst doldrums and into our lives as never before. The technology infrastructure that was laid down, and the tool chest that the training industry can now play with, is impressive.

VoIP companies like Skype and Vonage started a quiet revolution in the telecom industry, making free (or absurdly cheap) phone calls easily available to anyone with a web connection, and prompting pundits to forecast that 40% of all business calls will be web-based within three years. Microsoft came under pressure from all directions. Oracle acquired PeopleSoft for $10.3 billion, a move that surely made LMS vendors nervous.

Google bought Keyhole Corp, a digital mapping business, amid rumours that Google was developing its own browser to compete with Internet Explorer. IE lost double digit market share to the open-source Firefox browser (actually Netscape reincarnated) within weeks of its launch, a classic example of how fragile the status quo can be these days, and of how vulnerable even the most dominant are to innovative competitors. And open-source Linux went from strength to strength, landing a major endorsement when IBM ditched Microsoft and adopted Linux as its operating system of choice. That was before IBM sold its personal computer business to a company in China.

After Google’s massive IPO, Yahoo, Amazon.com and Microsoft all got the search engine bug, and lots of start-ups joined the search for better search systems. And the proliferation of spam and viruses made internet security a very good business to be in. Symantec acquired Veritas for $13.5 billion.

Apple, HP and Dell all rolled out desirable little devices that make the old data-focused grey-box notion of a personal computer redundant. The cell phone became a multimedia mobile computer that serves up music, movies, and images as well as the occasional phone call or text message. The iPod phenomenon made Apple the fastest growing stock of the year. In the US, broadband users outstripped dial-up users for the first time, making web-based multimedia more viable than ever. And wide-area WiFi hot-zones sprang up like mushrooms, with the city of Philadelphia announcing a plan to provide free broadband wireless access to everyone within the city limits.

There was less dramatic progress in social computing. True, new services kept popping up to provide “connections” for those seeking collaboration or just company. Friendster became phenomenally popular overnight, acting as a peer-to-peer social network for (mainly) adolescent gossip sharing. More mature versions have yet to take hold, though Spoke and the repositioned Lycos have promise. Free photo sharing services like Hello and Flickr have blossomed. And blogs and RSS finally made it to centre court, after years of being cult phenomena, suggesting that the emphasis in the application of computing has moved from the personal to the intimate. Perhaps training needs to start following the same path.

So what does it all mean for learning? Very little, if you choose to ignore it. Rather a lot, if you are looking to exploit it. Strangely enough, academic institutions are often more advanced in their incorporation of web technologies than business organisations. Perhaps it is because their primary market is the digital generation. It is hard to ignore the way your students prefer to do things, particularly if they make sense.

Universities are using social networks, WiFi, podcasting, online forums, and virtual collaboration almost as a matter of course these days. They are still semester-bound, which is odd given the inefficiencies of date-based curricula, but for the most part there is a buzz of innovation and experimentation on campuses around the world. Sadly, a similar innovative buzz is hard to find in the training centres of major corporations.

The oft-heard put-down of “solution looking for a problem” is very easy to use at the end of 2004. There are a lot of inexpensive, easily obtainable tools looking for someone to apply them. Technologies are no longer lagging our ability to use them. The reverse is true: our imagination and willingness to reinvent our processes and ourselves as trainers is the major limiting factor in our evolution to more competent professionals.

Ah, but is there a need to innovate, to continuously improve our effectiveness or efficiency, to provide a better service to our learners, and our employers? As an industry, the answer has to be a resounding “yes” but as individuals we tend to be happy waiting for someone else to make the first move. As Microsoft discovered at so many levels in 2004, playing a waiting game is not necessarily the smartest survival strategy.

Tuesday, January 04, 2005

New Year's Training Resolutions 2005

TrainingZONE Parkin Space column of 12/29/04:

Last December I made a list of “professional resolutions” that would make me altogether a better person in 2004. Having conveniently misplaced that list, I am not able to tell how I did, though I am sure it was not an admirable performance. Now it’s time to put together a new list. So what do I need to achieve in 2005 to do a better job as both a trainer and a consultant?

My New Year training resolutions:

1. Start learning Mandarin.

2. Do not get comfortable with one way of doing things, keep looking for better ways to achieve the goals, but do not be easily distracted by bright shiny learning objects.

3. Do not give up easily where decision-makers don’t want to be bothered with the complications of innovation. Try harder to justify the intelligent application of emerging technologies in learning designs where it is blindingly obvious (to me) that these will enhance learning. Develop better ways of explaining to non-technical traditionalists the use and impact of such tools as social networks, RSS, and peer-to-peer collaboration.

4. Work on a training evaluation system that actually measures my performance against the specified learning goals, and quantifies the impact that I am having on business objectives.

5. Stop acceding to demands for a training solution to what is really a management problem. I am sure that a large proportion of the training that takes place fits this mode, and it is not easy to turn away this kind of business, but I for one am not doing it any more. If I can’t successfully provide the consulting solution, I am not going to fake a training solution. Not only is it frustratingly inadequate to “train” people then send them back into an environment in which the training will have no effect, it abuses their time and energy, wastes the clients’ budget, and lowers the reputation of training generally.

6. Try to write an objective guide to SCORM and AICC in plain English that even I can understand. Focus on the practical implications for the design of learning experiences and clarify the rather specific niche that such standards and models have in the wider context of learning, and of learning administration.

7. Learn more from learners how best to help them learn. Spend more time with people to understand their evolving preferences and behaviours where learning is concerned. Try to gain better insights into their informal learning processes, and exploit those insights in more formal learning experiences.

8. Always put the needs of individual trainees ahead of my own needs, and the needs of my clients’ business ahead of all else.

9. Spend more time in the classroom and on the conference podium. Reverse my recent trend toward being very virtual, and get back to doing a lot more face-to-face communication. (For years I was a strong advocate of the virtues of virtual-ness, but, while the efficiencies are undeniable, there is a point beyond which your batteries stop recharging).

10. Campaign for free wireless broadband in every classroom, airport, conference venue, and hotel room. It is rapidly becoming as expected a part of the environment as air conditioning or electricity.

11. Design and build a prototype learning system that is delivered exclusively via low-spec mobile phone (I am particularly interested in basic literacy training).

12. Collaboratively engage with more people in the learning profession, with the goal of writing more useful how-to papers. Finish writing my book.

13. Do some pro bono work for a deserving cause.

14. Behave better: Stop using the fuzzy term “e-learning”. Stop rolling my eyes every time someone else uses it without clarifying specifically what they mean. Stop pronouncing it “HellMS” when in the company of LMS vendors.

15. Finally kick the PowerPoint habit.

It looks like 2005 will be another one of those NASA centrifuge-ride years. But I trust it will bring health, wealth, and happiness to all.

Sunday, January 02, 2005

LMS Adventures continued..

My LMS adventures piece a few weeks ago produced a torrent of email, most of it agreeing that today’s LMS has way too much say in our training decisions.

The only vaguely defensive reactions were from LMS vendors, who typically said “Yes, but what else can we do?” Short of reinventing your systems completely, you can probably do nothing. Corporate learning used to be controlled and channelled, but is increasingly informal and personal. If training is like an irrigation system, channelling knowledge to achieve preconceived ends, then raw knowledge is like rain – it will bypass an LMS that does not accommodate and focus it, and it will find its own way, carving a new learning landscape as it does.

There is a philosophical argument to the effect that learning cannot and should not be managed at all, but let’s not go there. I believe that formal and informal learning reinforce each other and that the absence of one impoverishes the other. Formal learning, almost by definition, requires “management” – it has to have some kind of strategic context, involve some kind of premeditated processes, and be organized or guided along in some way. So there is clearly a role for an LMS.

For those who thought I was being unduly unfair on the industry that gave us the e-learning supply chain, complete with its various intermediary gatekeepers and control systems, I apologise. I never intended to imply that a system for managing learning is unnecessary, merely that such systems today are inadequate. They are inadequate not because they provide too little control, but because they provide too much of the wrong kind of control, channelling the thinking of trainers away from creative innovation and toward dumb compliance with outmoded learning models. And they are way too precious about their uniqueness within the ERP family.

An LMS should manage access, record-keeping, and reporting on those things that require such administrative control. It should also enable the flawless provision of services to the customers of the learning department. An LMS should not dictate, or constrain, pedagogical architecture or learning strategy. Sadly, most do. How did this happen?

Because a simple adaptation of most ERP systems could accommodate the access and reporting aspects of learning management, LMS vendors needed to differentiate themselves from the big enterprise providers like SAP and PeopleSoft (now Oracle). So they focused on becoming ‘platforms’ for the launching of packaged content, which was outside of the brief of ERP providers. And that became their unique selling proposition and primary function.

Only the most cynical observer would suggest that the collusion between the LMS vendors and ADL in the creation – and aggressive promotion – of SCORM was deliberately geared to creating the kind of barriers to entry that result from getting an industry to adopt a proprietary standard instead of exploiting more open web systems. But that was the net effect. In standardising course interoperability around a content-centric object model, they effectively shut out the wider world of process-centric or distributed web interoperability. In so doing they denied to their customers access to some of the real learning potential of the Internet. Learning experiences facilitated by peer-2-peer systems, social networks, just-in-time mentoring, blogging or RSS? Just because your LMS cannot “manage” these does not mean they should not be experimented with and exploited.

SCORM advocates will of course deny that their reference model paints learning developers into a corner, or stunts their growth. But I challenge anyone to persuade a corporate e-learning decision-maker to try a product, solution, process, or experience that is not SCORM-conformant. If it is not LMS plug-and-playable, it is not taken seriously. And all too often you will be asked to dumb down your learning model till it fits the capabilities of the LMS, no matter how much you lose in learning effectiveness. Just say no.

The best bet for trainers is to accept that their LMS is only one tool, and a limited one at that. Use your LMS, by all means, for tasks for which it is appropriate. Exploit learning objects, in so far as they are relevant. But where your strategy and learning objectives call for an approach that takes you outside the scope of your LMS, do not pull back. Step boldly into that “unmanageable” territory. Be willing to use multiple learning processes and multiple management tools to achieve your learning objectives, even if it means compromising at first on administrative efficiency.

Letting your LMS be the primary determinant of the limits and scope of your learning endeavors is like refusing to eat anything that cannot be nuked in a microwave oven.

Bon appetit!

TrainingZONE Parkin Space column of 22/10/04