Getting Started With Learning Data and Analytics

Maybe there’s an executive demanding more visibility into your program, or perhaps you just want a better understanding of how things are going so you can find ways to make them better. You’re ready to get started with learning data and analytics, but where do you begin? Data is a powerful tool in the CLO’s toolkit, but figuring out how to start using it can be overwhelming.

There are two places to look for your first data analytics project. Look for what is easy or look for what is valuable. Do you have a learning system that is already producing data? Do you already have tools in your ecosystem that are experience API, or xAPI conformant? Is there a repository of data you can easily tap into? Those are easy places to start using analytics. Look for a low friction path to a quick win, even if the metrics don’t seem impactful yet.

Start using whatever data you can get your hands on and use it routinely. Some of the greatest value in learning analytics comes from incorporating data into your daily routine. Find a few metrics that are easy to monitor and incorporate them into your daily huddles or weekly staff meetings. Establish a baseline and watch how the metrics change over time. Bringing simple metrics into your consciousness will give you a much deeper understanding of the performance of programs and it gives early visibility into positive or negative changes happening in your business. Metrics can have the effect of gamifying your job and instilling a motivation and passion in your team.

Look for some particularly valuable analytics to gather, even if they’re not easy to come by. New programs or initiatives are opportunities to incorporate business-aligned metrics from the beginning. When launching a new program, take time to understand the business need behind it and find metrics that will suggest whether the learning program is having its intended impact.

Deploying a new learning platform or tool can be another opportunity to introduce analytics. Monitor metrics relating to its rollout to ensure it is being well-received and adding value commensurate with its purchase price. There are different levels of complexity and types of learning analytics to measure. It’s important to understand these continuums when planning a learning analytics journey.

Levels of Complexity

There is a continuum of sophistication when it comes to using analytics. Below are four levels of analytics complexity. These levels build on one another. You can’t get to the more sophisticated analytics without mastering the simpler levels first.

Measurement: Analytics starts with the simple act of measuring, or of translating information into data. Measurement is the simple act of tracking and recording values. At this stage, focus on generating clean data. We want to understand what we have, what we don’t have and what we can rely on.

Evaluation: Once we have data, we can begin to evaluate it to understand whether it means something good or bad. Basic evaluation is often called “descriptive analytics” because it describes what has happened. This level of analytics uses basic mathematical techniques (think middle or high school level math) along with charts and graphs to visualize what is going on in learning programs. The majority of learning analytics in use today are simple descriptive analytics.

Advanced Evaluation: Once there is an understanding of what has happened in the past, more sophisticated techniques can be used to understand why it happened. This level of analytics uses statistical techniques like correlation and regression analysis (think college level math) to determine which variables drive change in other variables. This is often called “diagnostic analytics” because it diagnoses why things are happening. This level of analysis often requires trained data scientists. It’s a journey where answering one question usually generates more questions. These advanced evaluation techniques are extremely powerful when trying to prove learning’s impact on a business metric that might be impacted by several other variables.

Predictive and Prescriptive Analytics: When datasets get big enough, they can start looking into the future. Predictive analytics attempt to answer the question: “Given what I know about the past, what is the most likely outcome if event X happens?” Prescriptive analytics goes one step further. It says: “Given that we know the most likely outcome of event X is going to be event Y, then when we observe event X, we should take action Z to optimize the subsequent result.” The ultimate application of prescriptive analytics in learning would be a sophisticated recommendation engine that delivers the right learning intervention in the right way at the right time to enable an employee to succeed in a task. This level of analytics requires powerful artificial intelligence algorithms and well understood data to be useful in a nontrivial way. It is rare for this level of data to exist in corporate learning today. Be skeptical of promises for AI for learning that go beyond trivial insights and applications.

Categories

People want to measure different aspects within learning. To understand the full scope of what is possible, it is helpful to examine three categories of learning analytics.

Learning Experience Analytics: When measuring learning experience analytics, we want to know about the content and classes provided to learners. Completion or utilization rates measure time spent in learning and when people are choosing to learn. We can dive into question item analysis, search term analysis or sentiment analysis on learner feedback. This category of learning analytics is all about understanding the learning materials.

Learner Analytics: This category seeks to understand more about the learners. Here, we look at competencies, skills gaps, learning patterns and preferences. This category often includes credentials and certifications that inform organizational readiness and compliance.

Learning Program Analytics: A learning program delivers a set of learning experiences to a set of learners with intention of driving a specific business outcome. Learning program analytics helps determine whether that business outcome was achieved. This is the most valuable form of learning analytics, as it forms the connection between learning and the business value that it drives. Before we can deliver reliable learning program analytics, a solid foundation of learning experience and learner analytics is needed.

Creating a Mindset With Data

Learning analytics is a journey that builds in complexity and sophistication over time. Don’t be intimidated by seeking perfection from the start. Management decisions are almost never made with perfect information, but rather, they deal in probabilities. Having some data is better than having no data. Be careful to understand its limitations and not overvalue the conclusions it suggests.

Start small and build learning analytics programs over time. Management isn’t used to seeing data come out of corporate learning, so it can be difficult to get support for early learning analytics efforts. It’s amazing how quickly an attitude can change once metrics are shown to speak to the business. Use early successes to get larger buy-in.

The real value in using data and analytics is the mindset that it creates. Data forces us to be accountable for delivering results. Data inspires to create a culture of continuous improvement. These culture shifts drive us to be better at our jobs and will make learning a strategic enabler of corporate success in the coming years.

This is the last of a three-part series on learning analytics. Read part one and part two here.

Listen to CLO Breakfast Club Podcast and analyze your approach to L&D.

Mike Rustici is founder and CEO of Watershed, a learning analytics platform that bridges the gap between training and performance. A 20-year veteran of the e-learning industry, he helped guide the first draft of Experience API, a learning interoperability standard. Comment below or email editor@CLOmedia.com.