More than a smile sheet

/

#8 in our training evaluation blog post series:

Digging into training evaluation uncovers a lot of debate and discussion around the value of level one evaluation data.

In my last evaluation post in this series, A little evaluation can be a dangerous thing, I wrote about the potential dangers of only using level 1 evaluation data to determine the effectiveness of learning back in the workplace. There are many articles, blog posts and forums dedicated to discussing the merits (or lack thereof) of using level 1 evaluation. I personally believe that a level 1 smile sheet has value to the learner as it allows them to reflect on their learning and provides a vehicle for their thoughts and feelings. But I also believe that we need to keep in mind that it’s only one small measurement in the overall evaluation process. Much less weight should be on the “qualitative” data gathered from a level 1 smile sheet and much more weight and importance be given to level 4 evaluation results - the impact training has on the business results.

Whether simple or complex, level 1 end-of-course evaluation forms (a.k.a. “smile sheets”) are used in the majority of training courses offered by organizations – in over 91% of organizations according to a 2009 ASTD Value of Evaluation research study. But does your level 1 end-of-course “smile sheet” go beyond the basic questions to capture data that will help your organization measure evaluation levels 2, 3 and 4?

A well-designed level 1 evaluation plan should measure not only learner satisfaction but also their level of engagement and relevance to their job. The goal is to incorporate statements or questions that focus the learner on higher levels of evaluation and get them thinking about how the new learning will benefit both them and the organization after the training event is over.

There are some simple changes you can make to your level 1 evaluation form that can provide further value:

  • Consider using a 7, 9, or 11 point rating scale to provide a richer level of feedback. Only label each end of the rating scale, rather than labeling each number on the scale (e.g., 1=strongly disagree and 7=strongly agree).     
  • Make all evaluation statements or questions learner-centred. For example, rather than “The instructor provided debrief activities for students to demonstrate their learning”, instead use “The debrief activities helped me to effectively practice what I learned”.        
  • Consider adding statements or questions to the course evaluation form that measure engagement and relevance. This helps to focus the learner on levels 2, 3 and 4. Some examples include:
    •  I had sufficient opportunities to contribute my ideas. (level 2)
    •  I estimate that I will apply the following percent of the knowledge/skills learned from this training directly to my job. (Provide a % scale from 0% to 100% in increments of 10.) (level 3)
    • This training will improve my job performance.(level 4)  

You can see that just a few tweaks to a level 1 evaluation leads to insightful information that can improve your training process.

Stay tuned for more upcoming blog posts with tips and strategies for other levels of evaluation and be sure to check out our other evaluation blog post in this series:

 

The 7 Principles for successful Succession Planning

/

#2 of 9 in our weekly succession planning blog post series:

Our guest blogger, Paul Riley is life-long learner of Organizational Leadership and Change who applies systems thinking and community development principles to help people work more effectively together within the complex human systems we create.

In my work, I've examined best practices among organizations that are known for succession planning, such as Boeing, Eli Lilly, Chase Manhattan, Southwest Airlines, and the U.S. Army, to name a few. After an extensive investigation, I came to the conclusion that there's no “best” way to manage succession. Although there's broad interest in defining processes and techniques for succession planning, there remains a considerable lack of consensus among the experts about how succession planning can be applied in a systematic, reliable, and consistent manner.

Because organizations have unique cultures and business contexts, which determine how talent is cultivated and nurtured, it's futile to prescribe one best method or strategy. So I’ve created a list of seven principles that can be applied to succession planning programs at any organization.

The principles are:

  1. Align Succession Planning Programs with the Organization’s Strategy
  2. Combine Succession Planning and Leadership Development
  3. Include All Levels of the Organization
  4. Create Opportunities for Practice, Feedback, and Reflection
  5. Promote Openness and Transparency
  6. Develop Simple, Flexible, and Decentralized Processes and Tools
  7. Continuously Monitor and Evaluate

The organizations that succeed at developing a sustainable leadership pipeline, approach succession planning as more than a process of updating a list. They prepare individuals for future roles, not just positions in the organization. Leaders must create long-term processes and programs for managing the talent pool across all levels, from the front line to the executive suite.  By investing time and resources to enhance organizational bench strength, organizations are able to address the realities of today’s complex business environment more nimbly and proactively.

Organizations often neglect succession planning due to lack of time and resources. We tend to focus on short-term results and managing the day-to-day operations, impacting our ability to engage in succession planning. However, neglecting to identify and develop successors can result in poor placement and promotion outcomes, which directly impacts productivity, motivation, engagement, and retention, because people are more likely to fail or leave if they are not properly prepared. Good talent management sets them up for success.

Want to know more about the Seven Principles of Succession Planning? Stay tuned for Part 3 of this series, which will discuss the first principle – Align Succession Planning Programs with the Organization’s Strategy.

Be sure to check out our other Succession Planning blog post in this series:

What’s so important about Succession Planning? 

 

A little evaluation can be a dangerous thing

/

#7 in our training evaluation blog post series:

I was reading an interesting article recently called, “Are you too nice to train?” by Sarah Boehle and she included an interesting case that I’d like to share:

Roger Chevalier, an author and former Director of Information and Certification for ISPI, joined the Century 21 organization as VP of Performance in 1995. The company trained approximately 20,000 new agents annually using more than 100 trainers in various U.S. locations. At the time, the real estate giant's only methods of evaluating this training's effectiveness and trainer performance were Level 1 smile sheets and Level 2 pre- and post-tests. When Chevalier assumed his role with the company, he was informed that a number of instructors were suspect based on Level 1 and 2 student feedback. Chevalier set out to change the system.

His team tracked graduates of each course based on number of listings, sales and commissions generated post-training (Level 4). These numbers were then cross-referenced to the office where the agents worked and the instructor who delivered their training. What did he find? A Century 21 trainer with some of the lowest Level I scores was responsible for the highest performance outcomes post-training, as measured by his graduates' productivity. That trainer, who was rated in the bottom third of all trainers by his students in Level I satisfaction evaluations, was found to be one of the most effective in terms of how his students performed during the first three months after they graduated. "There turned out to be very little correlation between Level I evaluations and how well people actually did when they reached the field," says Chevalier, now an independent performance consultant in California. "The problem is not with doing Level 1 and 2 evaluations; the problem is that too many organizations make decisions without the benefit of Level 3 and 4 results."

Industry studies appear to support his words. A 2009 ASTD Value of Evaluation research study found that 91.6% of the organizations in the study evaluated training at level 1, 80.8% at level 2, 54.6% at level 3 and 35.9% at level 4. 4.1% did no evaluation at all! Of the 91.6% that evaluate at level 1, only 35.9% said this level had high or very high value. Yet of the 36.9% of organizations that evaluated results (level 4), 75% said this level had high or very high value.

ASTD’s findings are somewhat alarming because they suggest that the majority of these organizations are going no further than level 1 evaluation, if they’re evaluating at all. We could assume from this data that the level 1 information gathered by these organizations’ training teams is the primary or maybe the only measurement used to justify their training efforts. Qualitative data and comments get rolled up into an overall total and used as a benchmark to measure the effectiveness of the trainers and the training programs being offered. Level 1 is used in isolation with no knowledge or thought about how the training programs address (or don’t address) key business needs. So why do companies do this?

I agree with Boehle’s theory that it comes down to two factors. First, Level 1 “smile sheets” are easy to do while levels 2, 3, 4 and 5 may appear to be costly, time consuming and potentially confusing (where do we start? How do we do it?). Secondly, if stakeholders (e.g. CXOs, internal clients and business partners) don’t demand accountability, why evaluate further? Digging in further may uncover negative results - if all appears to be working well on the surface, no one is asking questions and learners are happy, why rock the boat?

It’s been our experience that the best practice training and development teams recognize that they have a responsibility to ensure that the programs they produce and deliver are aligned with the organization’s needs – to demonstrate how training is contributing to the success of the organization. They need to show proof that training is really making a difference - clearly identifying how organization’s bottom line is being positively impacted and how business needs and issues are being addressed. Using only level 1 data to measure training and trainer effectiveness is dangerous and tells very little about how much learning is actually taking place on the job and how business results are truly being impacted. And sooner or later, this will catch up to the training providers and ultimately to the organizations they work for. Training budgets will be cut, work will be outsourced, and organizations will struggle to keep up with their competition in a tight and highly competitive economy.

Be sure to check out our other evaluation blog post in this series:

What's so important about Succession Planning?

/

#1 of 9 in our weekly Succession Planning blog post series:

Our guest blogger, Paul Riley is life-long learner of Organizational Leadership and Change who applies systems thinking and community development principles to help people work more effectively together within the complex human systems we create.

Succession planning has become a hot topic lately as organizations are preparing for a mass exodus of baby boomers from the workforce. Their impending retirement is already having a major impact on workforce capacity. Vacancies in senior positions are on the rise, and demographics indicate that there are statistically fewer people available to replace them. But there are more reasons that succession planning should be a part of your organization’s strategy.

The most important reason is that we rely on staff to carry out our missions, provide services and products, and meet our organizations’ goals. What would happen to your mission and organizational goals if key staff were to leave? How would your ability to deliver services and products be impacted? Succession planning helps to ensure that your organization has the right people in the right positions at the right time.

Although employers often use words like “loyalty” and “stability” to recruit quality candidates, employees understand that nothing is carved in stone with regard to their future employment with any one particular employer. The company’s bottom line often takes priority over loyalty when it comes to employment decisions. For this reason, there’s no longer a stigma associated with “employer-hopping.” In fact, we’re now referring to hopping around as a “portfolio career,” which sounds much more intentional and strategic. Employees can take their skills and knowledge and match their portfolios to a particular employer’s needs at any given time. Because workers are more mobile, organizations are competing to recruit and keep talented employees. Succession planning helps employers identify, develop, and retain top performers.

Due to the growth of the global market and global competition, as well as the recent economic crisis, companies are restructuring more frequently and more rapidly than ever before. This creates gaps in leadership at all levels of organizations. Therefore, organizations need to move away from traditional “management succession planning” models that are designed to meet promotion needs and often neglect to address leadership gaps at the middle and lower levels of the organization. Instead, we should focus on “technical succession planning,” by creating processes and systems to transfer specialized knowledge and experience at all levels of the organization. Supporting knowledge transfer perpetuates the collective corporate culture, institutional memory, and tacit knowledge within the organization, which are key ingredients for a sustainable future.

To learn more about how you can implement an effective succession planning program in your organization stay tuned for # 2 in this series – The 7 principles of successful Succession Planning.

 

Are YOU learning?

/

Stephanie and I recently came across a request for proposal that asked for proof of our recent professional development and it got me thinking.  Although we’re in the business of creating and providing learning opportunities for our customers and clients, many of us neglect our own ongoing self-development and learning.  Granted, we’re all busy with looming deadlines, multiple projects and countless day-to-day tasks but we get so focused on these activities that we sometimes forget to lift our heads up and focus on the bigger picture.  I’ve been guilty of that myself.  But how can we provide relevant and current learning solutions to our clients and customers if we’re not updating our own knowledge and skill?  

Stephanie and I strongly believe that educating ourselves is a priority in our profession and we constantly work on doing just that.  We read books and research reports, contribute to on-line discussions, complete certification training and many other things to keep up with what’s going on in the industry.  Doing a Google search uncovers an incredible amount of informative and thought-provoking T&D information. Books, research and best practice articles, information, discussions and debates on blogs, LinkedIn groups and Twitter feeds as well as meet up groups, conferences, online webinars and podcasts are available and most are free! As an example, I recently started using Twitter (@PaulaYunker) and am amazed at the amount of information that “Tweeters” in the industry are sharing. Through their tweets, I’ve found new inspiring blogs to read, informative research, the latest T&D trends and practical tools and tips for instructional design. I’ve also taken advantage of a number of free webinars and am currently reading a great book that was recommended by others online.

So make a conscious decision to “educate” yourself on a regular basis. Schedule an allotted amount of time each week into your calendar and stick to it.  With a little time and practice it will become a habit - just another one of the tasks you accomplish each week.  It’ll benefit not only you but your clients and customers as well.

So, ask yourself…what are you learning this week?

 

UPDATE - Free Summer webinars (July and August)

/

Here’s an update to the Free Summer webinars blog article I posted in late June.  The following are a few additional webinars that didn’t make my original list that sound interesting. 

JULY

Friday, July 15, 10am-11am PST: Ubiquitous Learning: Next Generation Solutions for Learning Anytime, Anywhere in Any Modality

This concept taps into the reality we've been working on that allows a learner seamlessly flow through the learning experience from desktop to smartphone to tablet to TV whenever the time or mood suits them. Apps are now available on virtually every electronic device we touch and use in our daily lives, and cloud computing models enable the ability to synchronize and manage the learning experience on whatever device best suits our environment and mood.

Thursday, July 28, 10am-11am PST: Mobile Learning Basics + (Free) Mobile Learning guide

The Academic Advanced Distributed Learning (ADL) mobile learning team recently released a mobile version of their mLearning guide on the web and in various platform stores.  This webinar will discuss the single development for deployment on multiple devices, the choices made and the lessons learning.

Thursday, July 28, 10am-11am PST: Driving Development and Growth with Performance Management

Join Stacey Harris, VP of Research for the Brandon Hall Group, as she shares business case examples of how organizations are driving cultures of development and performance, by focusing more on the human connections and less on process compliance. She’ll discuss the impact of enabling a culture of coaching and feedback models both on engagement and performance, and how organization are leveraging individual development plans as part of their strategic talent planning tools.

 

AUGUST

Wednesday, August 10, 10am – 11am PST: Mentoring for the 2020 Workplace

Forward-thinking companies are reinventing the meaning of mentoring and using innovative techniques to be able to scale in mass proportions to prepare for the future workplace. Based on research from the book "The 2020 Workplace" by Jeanne Meister and Karie Willyerd, this session will cover examples from companies like American Express, AT&T, and PricewaterhouseCoopers to explore how to get talent ready fast for the upcoming shifts in the workplace.

Wednesday, August 17, 10am-11am PST: Gadgets, Games and Google for Learning: How to Leverage the Latest Technologies for Learning and Performance Improvement

This session highlights how organizations are leveraging Smartphones for performance support and mobile learning, how game-based learning is being designed to teach everyone from firefighters to sales reps to call center employees and how quick searches and meta-data are changing the landscape of how employees learn and their information expectations. Join Karl Kapp, author of four books on the topic of the convergence of learning, technology and work as he explores some of the research, thinking and examples of learning technology in action.

Tuesday, August 23, 10am-11am: An Active Approach to Leadership Training

Research has shown that experiential learning is one of the most effective ways to keep participants engaged and active throughout the learning process. This webinar will explore a variety of high-impact activities, jolts, and interactive learning strategies that have been specifically chosen to teach and apply some of the most critical leadership competencies our evolving leaders need.

Tuesday, August 23, 10am-11am: The New Role of the LCMS: Enabling New Ways of Learning

Richard Nantel, co-chief executive officer at Brandon Hall Group, will discuss how today's LCMSs allow content authors and subject matter experts to collaborate on learning content and publish it to today's various computer and mobile devices. Jeff Whitney, vice president, worldwide marketing, Outstart, will describe best practices for how LCMSs are being used innovatively in real-world situations.

Also check out an earlier blog post of other free summer webinars.

 

How much to evaluate?

/

#6 in our training evaluation blog post series:

I read an article the other day called Measuring the Effectiveness of Training, by Teresa Hiatt, Director of Sales Education, Ricoh Americas Corporation. She stated some interesting statistics in her article that I’ve also run across in other research articles and best industry reports:

  • Research from the American Society for Training and Development (ASTD) states that top companies will spend more than $1,500 per employee per year for education opportunities.       
  • Fortune Magazine's 100 Best Companies to Work For lists development opportunities as an important indicator to attract and retain top employees.
  • The latest estimate of the size of the Training Industry is about $135 billion.

She concludes that all this research confirms that leading organizations and key management think that training employees is an important part of business today. Yet despite the time, effort and money companies spend for training and development, training is rarely held accountable for results.

Many organizations (and even T&D professionals) are under a misconception that measuring levels 3 and 4 (and even 5) are complicated, require too many resources and cost too much money so they just don't do it. They also believe that the levels of evaluation should be applied equally to all courses they offer. But not every training program requires evaluation at all levels. Business needs, strategic positioning, performance discrepancies, stakeholder support and the ability/willingness of the organization to observe, document, evaluate and support evaluation within an organization are all important factors when determining what training programs are chosen for evaluation. 

Best practice organizations use the following recommended evaluation targets when conducting evaluation of training programs:

Level 1 = Conduct evaluation for 100% of training programs

Level 2 = Conduct evaluation for 60% of training programs

Level 3 = Conduct evaluation for 30% of training programs

Level 4 = Conduct evaluation for 10% of training programs

Level 5 = Conduct evaluation for 5% of training programs

What percent of your training programs are you evaluating and at what level(s)? I’d be interested to hear.

Be sure to check out our other evaluation blog posts in this series:

Selling the importance of evaluation

/

#5 in our training evaluation blog post series:

In the not so distant past, evaluation of learning was an isolated activity relegated to the training team who’s responsibility didn’t extend much past gathering level 1 and level 2 evaluation. Today the emphasis is on the bottom line and how organizations can get the best value for their money and efforts. New evaluation tools, processes and strategies are available to help companies become more strategic; the evaluation of learning has become less of an isolated activity and more of a culture/philosophy. Learning teams are now becoming drivers of change, helping to support evaluation efforts within their organizations. But what if your stakeholders and senior management don’t see or understand the importance of evaluation?

Implementing levels 3, 4 and even 5 can be a challenging and daunting task even if the training team is fully involved and committed, because commitment and participation is also required from employees, managers, supervisors, business partners, stakeholders and senior management. An organization’s executives need to be on board as top-down messaging is critical to success – you’ll be swimming upstream trying to get managers to participate in evaluation if they don’t feel that their own bosses are behind it.

Many senior leaders already acknowledge that employee education is a critical success factor for future growth and prosperity. Use this as your ‘hook’ to sell them on the importance and value of a solid system of evaluation. Here are some tips to help you along the way:

  • Show how evaluation contributes to success: Be able to show a direct correlation between the organization’s strategic needs and goals, business unit operational needs, individual development needs, the training that is designed to address these and the evaluation techniques that will be used to quantify improvements.
  • Share a roadmap to implementation: Create an evaluation strategy that will systematically guide the organization from the present situation to the desired amount of evaluation. Be prepared to provide costs in terms of time and manpower.
  • Give confidence with examples: Gather relevant case studies of best practice organizations who have implemented evaluation within their organization with positive results. Use this information to support your position. There are a number of websites with best practice research. Check out our favourites in a previous blog post.
  • Start small to prove your case: Run a pilot and communicate/share results. Work with a key stakeholder/business partner to address a business need through training. Use this training as the “test case” for your evaluation plan. Apply each level of evaluation gathering testimonials and data, and tracking trends along the way. Share results and testimonials. Use stories and case studies based on the training results to capture attention and highlight the positives. Prove that measuring the value of learning can have a positive effect on your organization.
  • Communicate, communicate, communicate! Communication at all levels of the organization is critical to success. People need to understand the What, Where, Why, When and How of your evaluation strategy and what their role and commitment will be. Be patient. This will take time, but your efforts will be worthwhile!

Be sure to check out our other evaluation blog post in this series: