Meaningful training analytics: 1+2 ≠ 4

When the pandemic began, Susan was tasked with creating a remote leadership program for her company’s managers. She studied techniques, conducted virtual training sessions and gave the leaders as many good ideas as she could. Ninety days later she sent them a survey and asked how they were doing. The response rate was low because the managers, still working remotely, were distracted and busy.

Six months later, Susan was asked to report the data from her training. She didn’t have much to show except the training platform’s interaction report, the post-training survey results and the limited 90-day survey results she received. She tried to make a connection between the training and some of the company successes, such as high employee retention. Executives were not impressed, and they decided that training could be outsourced to save money.

This familiar story is occurring globally. Even some award-winning training programs are missing the critical element that will ensure program success and demonstrate organizational value: Level 3 data connecting on-the-job performance to program results.

Delivering training and then looking for the results leads to skepticism about the value of professional development. The missing link — Level 3: Behavior, in The Kirkpatrick Model — is where the value of training is created so the desired results are realized. The prevailing belief that a Level 3 plan for post-training support and accountability is difficult, expensive and out of training’s purview is untrue. Here are the deceptively simple steps to creating training with true value.

The end is the beginning

Begin designing your program using The Kirkpatrick Model, known as the four levels of training evaluation. When you receive a training request, ask what Level 4 Result it will support. In other words, what high-level organizational outcome will the training positively influence? For example, will it reduce employee turnover? Increase sales or productivity? Reduce waste or scrap? Increase customer satisfaction?

From there, design the piece that is so often missing: your Level 3 implementation plan. Training and the business should work together to define what needs to change in the on-the-job environment and employee performance to yield the desired results.

Then, the training professional can design the training program and related materials (Level 2) with a strong focus on what happens after formal training is complete. During this process, they can consider the environment that would best support the learning effort and eliminate distractions from learning and practicing the new knowledge and skills (Level 1). This transforms training from an event to a holistic, ongoing process.

“It might sound like a cliché, but we execute training exactly according to The Kirkpatrick Model, and it works,” says Josh Henshaw, director of talent development for Lumen. “During our scoping conversation with business leaders, we identify the business outcomes for Level 4 measurements and then immediately consider the Level 3 skills and behavior changes needed.”

Level 3 — where training value is created

One plus two does not equal four, meaning that formal training alone, no matter how good, will not produce a meaningful level of organizational results and cannot be credited with results without sufficient data showing the connection.

There is strong agreement that formal training alone yields a small portion of the results for any program. On-the-job experience, or learning in the flow of work, is one of the biggest sources of learning for employees. The on-the-job environment and culture significantly impact what employees will do, regardless of their knowledge.

Training budgets are often cut because data connecting learning to organizational outcomes is missing. Therefore, creating a Level 3 plan and defining roles and responsibilities early in the planning phase is critical not only for the success of the program but for the continued existence of the training function as we know it today.

Define critical behaviors

The first step in creating a Level 3 plan is to define the critical behaviors that learners should perform after training. These should be expressed in observable, measurable terms that a typical person can identify and understand.

For example, if you want team members to communicate effectively, a critical behavior might be, “Discussions are conducted in two-way conversations instead of in writing.”

Even when people know what they are supposed to do, many factors influence their behavior. For example, most people know that discussing complex issues via email is time-consuming and long email strings can be frustrating and confusing. However, people do it to avoid having a meeting or through a misguided belief that it will save time.

Create a support and accountability package

For important initiatives, the Level 3 plan should include a variety of methods and tools to support people in performing critical behaviors and holding them accountable for doing so. Providing support can be quite fun. Look for ways to help people do the right thing, such as job aids, checklists, small rewards and reminders. If resources are available for formal coaching and mentoring, these can be effective elements.

Holding people accountable is generally less fun, but it is usually necessary for a program to produce results. Think about techniques that are palatable in your organization. For example, you can have a dashboard that shows each person’s progress to create friendly competition. Or supervisors can privately track performance and discuss it individually with direct reports if that is more comfortable.

“Improving any size organization must address and support the desired changes in behavior to ensure employees are competently applying what they learned during training,” says Dr. John Plifka, director, Army Quality Assurance Program. “This is so critical for the U.S. Army considering our mission(s) and why we use a wide span of methods ranging from individual coaching, counseling and mentoring to evaluating the institution by conducting accreditations with a three-, six- and 12-month external evaluation follow-up.”

Monitoring performance takes time and resources. During program development, discuss post-training roles and responsibilities and obtain buy-in for a plan. If resources are tight, look at methods of self-monitoring and reporting and peer mentoring as viable alternatives.

“The Lumen call center new-hire training program increased the speed to competency and shortened the time to full production-level performance from 90 days post training to 30 days post training because we used Level 3 data to evaluate the effectiveness of training on critical role-specific skills,” Henshaw says.

The degree to which the Level 3 plan is implemented defines the success of the initiative. With this in mind, we highly recommend that the post-training support and accountability are defined as part of the training program and built at the same time as formal training materials.

Tips for a successful training plan

Training analytics are often summative in nature, meaning they occur at the completion of a training program or implementation phase. Shift your focus to formative evaluation, or that which occurs during a phase, so you can correct issues and maximize performance and results along the way.

Before training, find out what analytics are important to the stakeholders, managers of training graduates, training graduates themselves and the training group. Determine who will gather and report the data and in what format. Decide which data will be reported throughout the initiative — and when — to track progress and identify areas for improvement.

Let training participants know before and during training what they are expected to do after training, what support is available to them, how their performance and outcomes will be tracked, and how they will contribute to organizational success. Making the connection between training, performance and high-level outcomes has many benefits. Participants come to training with a higher level of interest and are more engaged. They learn more. Applying their new learnings on the job may not be easy, so remind them of the support available to them and the importance of their success.

Perhaps the most critical time for creating training success after formal training is when training graduates return to the job. Ensure there are regular checkpoints to verify that the reinforcement, support and accountability actions are occurring. Use technology to set automated reminders for yourself, supervisors, stakeholders and training graduates.
Most programs are not instantly successful. Expect to have setbacks and be prepared with a remediation plan. This is the reason that continual monitoring and reinforcement is required for success in most programs. Don’t consider changes in the plan to be failures. They are an expected part of the process.

Reporting on Level 3 analytics

Powerful data are a combination of numbers and supporting information. This formula connects what is learned, to what is done on the job, to the results it creates. If you attempt to report program value with numbers alone and no stories or testimonials, there is typically the objection that some other factor created or influenced the results.

There are always several factors that influence results, such as training, economic conditions, workload, coaching, marketing and promotions, and countless other conditions. With a reasonable investment of resources, you can show the relative contribution of multiple factors and create a culture of collaborative success.

A simple way to connect training to results is to survey training graduates and their supervisors on what factors contributed to their success. For example, supervisors could be asked what factors contributed to their success as a team leader.

Responses could include:

• Skills learned in leadership development training
• Company culture and values
• Regular schedule of team meetings
• My personal interest in my direct reports
• Support from human resources
• Technology and tools available to employees
• Company reward systems
• Peer mentoring support

Then, the supervisors can be asked to share a story of how one or more factors contributed to their success. The responses to these questions show the value that training brings to the company overall, as well as highlight other important factors in company success, giving credit where credit is due and contributing to a team-based approach to success.

Getting started

If you have never created a Level 3 plan, don’t be intimidated. Select one important initiative and use it as a pilot rather than trying to implement a new process concurrently for all your major programs. See if you can find an executive sponsor who will support the plan and discuss this article with them. Create a committee to work with you. Focus more resources on the post-training plan than on the training itself.

The time you spend during the pilot is an investment that will make subsequent programs faster and easier. Before you know it, you will be building and implementing Level 3 plans for all important company initiatives and successfully connecting training, performance and results.