Measurement: The Thorn in Learning’s Paw

In all of my years in the learning industry, I have seen no issue that creates more passion than learning measurement. In a recent conversation with David Vance, the issue once again moved to the top of the discussion. He and I agreed that our experience working with corporate learning leaders has revealed the divide over and over again.

There are two widely divided camps on the issue. In the first camp are those who see a need to measure learning. In the other camp are the traditionalists who see no need to measure. This is likely the largest group of learning leaders, and it is certainly the one most widely embedded in corporate learning’s heritage.

Like an iceberg, there are the reasons to measure that are more visible above the water line as well as a large mass of reasons not so visible below the water. One above the water line reason is that an organization’s culture doesn’t require a rigorous financial measurement of outcomes or return on investment. The iconic example of such an organization is Qualcomm, where Tamar Elkeles led learning initiatives for years. The 2015Chief Learning OfficerLearningElite award winner’s announcement captures the root of the learning culture at Qualcomm: “Learning has been a part of Qualcomm’s founding DNA since the 1980s.”

A stronger cultural statement would be difficult to find. Elkeles has been a passionate and outspoken advocate of the not-to-measure camp. The implication is that resources committed to measure would be redundant and wasteful. I get that, and if I were a learning leader in such an organization, I probably would take the same position.

But Qualcomm has been a unique enterprise — one where intellectual property is the ultimate corporate value creator and where the strategic importance of learning is fundamentally understood and is an important strand in the organization’s DNA.

The problem for the vast majority of chief learning officers and learners is that intellectual property is a support asset, not a core asset, in the business model — one that aids value creation activity and is not aprimary revenue and profit driver. In these organizations, learning is often viewed as overhead, an expense to be minimized. Resources are allocated largely based on financial considerations, not DNA. In these organizations, I argue, it is more important than ever to measure, to show the C-suite the financial return on learning investments. In these organizations, the answer is “to measure.”

So, if it is so important to measure financial impact, why do so few do it? At the risk of agitating some of my colleagues, I am going to get a bit personal here. I believe we sometimes avoid measurement because:

• Asymmetry of risk:We fear that the measured result might not validate the original learning effort, and that this might lead to suspicion about the learning program itself. To the sponsoring learning executive, this risk looks like it’s all downside with little to no upside potential. Also missing is the C-suite’s concurrent commitment that learning initiatives with attractive financial returns will receive adequate funding required. To my learning colleagues, I assure you that I understand the risk asymmetry. So, as a risk management prototype, do financial measurement on a small initiative, then scale the effort as the financial results are produced.
• Absence of know-how:I believe the best way to measure financial impact is with an experimental design that has an intervention group — the learners — and a propensity scored matching control group. The analysis is done with multivariate statistical analysis. But, most learning leaders are not trained in experimental design and statistical analysis, and a natural discomfort is often the result.

In the end, the question of whether to measure learning has much more to do with organizational culture and individual behavior than it does with financial analysis and return on investment. Despite this reality, designing the experiment and doing the analysis is more important than ever. Financial ROI is the language of the C-suite, and executing it well provides the CLO with a golden ticket to that coveted seat at the table.