MTA: Why the Kirkpatrick Model Works for Us

The MTA, a division of the Maryland Department of Transportation, is one of the largest transit systems in the United States, and its buses, light rail, subway and paratransit serve riders in Baltimore and throughout the state of Maryland. In early 2012, the MTA had a problem — too many of its recently trained bus drivers were involved in accidents.

That summer, Michael Wiedecker, a 25-year veteran of the MTA, was appointed director of operations training, which meant this became his problem to solve. He and the training team reviewed hiring procedures and new bus operator training and found no obvious flaws. Course materials, classroom and in-vehicle instruction, even the instructors were all largely unchanged from the previous years when accident rates were lower. Fortunately, as 2013 approached, hiring slowed, giving Wiedecker and his team time to find a solution.

As he settled into his new job, Wiedecker read Jim and Wendy Kirkpatrick’s book, “Training on Trial,” which inspired him to implement the Kirkpatrick training evaluation model at the MTA. The book provided a blueprint to build a chain of evidence so he could link the training team’s efforts to business results.

The four levels of training evaluation Don Kirkpatrick put forth first in the 1950s are well known to learning leaders. According to the 2016 ATD report, “Evaluating Learning: Getting to Measurements that Matter,” nearly 90 percent of organizations implement Level 1: reaction — participants’ feelings about the training; 83 percent also measure Level 2: learning — the gain in knowledge or skills due to training. Sixty percent evaluate Level 3: behavior — how participants apply training on the job. However, only 35 percent of surveyed organizations measure Level 4: results — the impact training has on the organization as a whole.

Implementing the Kirkpatrick Model

“Evaluating at only Levels 1 and 2 is not associated with higher on-the-job performance or business effectiveness. To really improve major learning programs and their impact on the business, organizations need to implement all four levels, with a focus on Levels 3 and 4,” said Wendy Kirkpatrick, president of Kirkpatrick Partners, and a contributor to the report. In 2012, the MTA had only Level 1 and 2 evaluations in place.

The methodology to implement the Kirkpatrick Model is straightforward. Essentially, start with the business goals, Level 4, then work down to Level 1. Along the way, determine which metrics are needed at each level to show training is effective and the procedures necessary to collect these metrics. For learning leaders who focus nearly entirely on classroom instruction, Levels 3 and 4 can be unfamiliar. One of the first things Wiedecker did was to get himself and his team certified on the Kirkpatrick Model. “We needed a common language so we could work together developing our evaluation methodology and redesigning our courses.”

For new bus operator training, Level 4 was obvious; the organization needed to reduce the number of accidents. “What was not obvious was where the data was to track this,” Wiedecker said. However, it turned out much of the data was already in the MTA learning management system. After a driver has an accident, they have to attend post-accident training; this along with new hire training results are recorded in the LMS. The MTA had a contract with PTG International, a local consultancy that specializes in training performance evaluation. Under Wiedecker’s direction, PTG built an interactive dashboard to display new bus operator performance 3, 6 and 12 months after completing training.

To evaluate Level 3, Wiedecker relied on instructors who had been drivers to identify behaviors critical to safe bus operation. Elmer Coppage started at the MTA in 1990 as a bus operator, became an instructor, and is now the manager of training and development. For the Level 3 observational assessments, he and his colleagues identified more than 60 behaviors needed to ensure candidates drove the bus safely, such as “keep both hands on the wheel” and “look left, then right, then left at intersection.” After graduating from the 10-week course, new operators enter a 90-day probationary period. In those 90-days, an instructor may board their bus and evaluate their performance at any time.

Pay Close Attention

One of the biggest changes in the new hire training was around focus. With a goal to reduce accidents, Level 2 evaluations, both written exams and observational assessments, became more rigorous. Prior to implementing the evaluation model, the goal for new-hire training was to fill bus operator vacancies, and graduation rates were close to 100 percent. Getting new drivers on the road meant less overtime and more reliable service, which was fine, until the number of accidents rose.

Level 1 evaluations measure learner’s satisfaction with the instructor, course content and training environment. Inspired by Wiedecker’s leadership and the MTA investing in their skills with the Kirkpatrick certification and additional professional development, Coppage and his instructors started looking to improve all aspects of the course. To make the classroom more engaging, they replaced projectors with large-screen monitors and added polling devices to make presentations more interactive. Coppage installed GoPro cameras on training buses, and recorded videos let candidates evaluate their own driving.

The MTA is committed to providing service to customers with disabilities. To elevate this portion of the training, Coppage found and refurbished three electronic wheel chairs. During the program, each candidate spends a full day in one of the chairs, which includes catching a bus into town and a different one back to the training facility, so they truly understand these customers’ challenges. “Implementing the Kirkpatrick Model helped us align the training program to the MTA’s mission, prioritizing not only safety, but also customer service,” Coppage explained. “We train our new operators to be professionals and to think of their bus as their office.”

So, what caused the rise in new bus operator accidents? Like most organizations, employee attrition is fairly constant, but hiring is limited to periods when budget allows. In 2012, the MTA was able to back fill a number of open bus operator positions, doubling class size. The new hires did not get adequate time with instructors in the classroom nor on the road to build the skills to recognize and avoid potential accident scenarios. With the dashboard in place, Wiedecker and Coppage deduced the student-instructor ratio was a key factor for success in training new bus operators.

In 2015, the MTA ramped up hiring again, this time quadrupling class size. But with dashboard data, Wiedecker and Coppage were ready. With their chain of evidence, they successfully made the case that even though the new hire course had improved, time with instructors was still critical to success. To meet hiring goals, they decided to use union instructors. They identified 30 experienced bus drivers and spent sufficient time training them on course materials and how to instruct adults. This approach worked well. Even with the large number of new operators, accidents have continued to decline, and the cost per accident is declining even faster. Total claims for 2015 were $560,000, less than one-fifth of the $3 million paid out in 2012.

The Kirkpatrick Model provided a framework for success at the MTA. The training team is now demonstrating real value to the organization, helping save millions by reducing accidents while improving customer satisfaction. Success with this project may soon be replicated with invoice processing, which will be accelerated courtesy of a training program now under development.

While many learning teams struggle to evaluate Level 4 results, Wiedecker said “It’s easy; we really want to be business partners with each department. I ask them what their business goals are, and we find a way for training to help them reach those goals. What department head would not want our help in achieving their goals?”

Charles DeNault is senior analyst, talent management for Wainhouse Research. Comment below, or email editor@CLOmedia.com.