Turn Diversity Evaluation Into Results

Evaluation is a task that every diversity practitioner will face. No matter your role, conducting an evaluation to assess key aspects of your diversity and inclusion initiatives is inevitable.

People do not always agree on one definition of evaluation. Here are two different definitions:

• The systematic process of collecting and analyzing data in order to determine whether and to what degree objectives have been or are being achieved.
• The systematic process of collecting and analyzing data in order to make a decision.

The first half of each definition is the same, but the reasons for collecting and analyzing the data reflect a notable philosophic difference behind each definition. The first reflects a philosophy that as an evaluator, you are interested in knowing only if something worked, if it was effective in doing what it was supposed to do. The second statement reflects the philosophy that evaluation makes claims on the value of something in relation to the overall operation of a diversity program, project or event.

Many experts agree that an evaluation should not only assess program results but also identify ways to improve the program being evaluated. A diversity program or initiative may be effective but of limited value to the client or sponsor. You can imagine using an evaluation to make a decision (the second definition) even if a program has reached its objectives (the first definition). For example, for nonprofits, federal grants are based on whether the program has achieved its objectives, but the harder decision to downsize or change may be a consequence of the second definition.

For some, endorsing diversity evaluation is a lot like endorsing regular visits to the dentist. People are quick to endorse both activities, but when it comes to doing either one, many diversity practitioners are very uncomfortable. I want to demystify important aspects of designing and conducting a diversity ROI evaluation by helping you get to know a few diversity metrics processes that matter.

All organizations possess data and information that could help evaluate a diversity program or project. These data are the one thing that all evaluations have in common regardless of the definition: “Evaluation is the systematic process of collecting data that help identify the strengths and weaknesses of a program or project. The data may be as simple as records of attendance at training sessions” or, as complex as “showing test scores showing the impact of a new educational program on increasing students’ knowledge across an entire school system.”

Evaluating Efficiency, Effectiveness, and Impact
We can define diversity evaluation even more closely as a process, which is guided by the reason for the evaluation. The evaluation might examine a diversity-training program, in light of values or standards, to make certain decisions about the efficiency, effectiveness or impact of the program. To understand the concepts of each, think of these three terms as the levels of a program evaluation.

Efficiency
Efficiency relates to an analysis of the costs (i.e., dollars, people, time, facilities, materials) that are expended as part of a program in comparison to either their benefits or effectiveness. How is efficiency, or the competence with which a program is carried out, measured in a program? Diversity practitioners would look at the efficiency of how details are carried out in a program. Diversity programs and initiatives often begin with recruiting, gathering materials, providing for space and setting up fiscal procedures. The relationship between the costs and end products becomes the focus of an efficiency evaluation. Although very important, these aspects of efficiency have no bearing on the program’s effectiveness. If the investment in the program or project exceeds the returns, there may be little or no efficiency.

Let’s consider an assembly line facility that houses a substantial training and staff development department. As part of this department, 10 instructors are responsible for ensuring that 500 employees are cycled through some type of diversity training every six months for a minimum of 20 hours of training each cycle. The training revisits the employees’ basic knowledge of their job and introduces new concepts of diversity that build additional competencies since the last training. The staff development department might work very efficiently by making sure that all employees cycle through in a timely fashion and in small enough groups to utilize the best of what we know about how adults learn. The students’ time on task is often not enough, however, and many of them do not retain much of what was covered in the training. Thus, the program is not effective.

Effectiveness
When you examine the effectiveness of your diversity and inclusion initiatives, you are asking this question: Did the activities do what they were supposed to do? Simply put, an initiative’s effectiveness is measured in terms of substantive changes in knowledge, attitudes or skills on the part of the program participants.

Although the right number of participants may have been recruited and the best possible site may have been secured, the effectiveness test is this: Did the activities provide the skills to effectively handle diversity and inclusion-related situations? Did the participants gain the knowledge they need to work across generational differences?

Here is an example of training that was effective, but had little impact on changing the employee’s behavior. The same diversity and inclusion department staff may conduct a training session on a new approach to de-escalate cultural conflict situations. The trainer may test all participants as training begins, and then test again once training is completed. The results are compared to determine whether participants’ knowledge increased, decreased or stayed the same. A knowledge increase would be an indication that the training was effective. Yet two weeks after the training, when one of the employees was back at her job location, a situation arose in which she engaged in a serious altercation with another employee and failed to use the skills taught. She used her older, more comfortable procedure for addressing communication differences across cultures and caused a problem that put her and her co-workers at the risk of suspension.

Impact
Thus the program impact on the people or organization becomes an important evaluation consideration. Impact evaluation examines whether and to what extent there are long-term and sustained changes in a target population. Has the program or initiative brought about the desired changes: Are employees using the new procedures? Do your employees have more job satisfaction?

Evaluators frequently pay too little attention to assessing impact. One reason is that “impacts” often manifest themselves over time, and diversity practitioners have already turned their attention elsewhere before computing this aspect of the evaluation. The actual impact that new procedure training might have in people’s everyday life often needs time to percolate and evolve. An attempt to collect impact data after allowing for this delay may run into a number of roadblocks such as learner turnover (you cannot find them), job or circumstance change (they no longer are in situations that demand heavy use of the skill taught) or lack of time or resources for the evaluator to conduct a follow-up.

Still, program and project sponsors are most interested in impacts. Whether a learner feels satisfied with the training or the training results in knowledge gain means little to a sponsor or employer if the learning doesn’t help the organization.

Evaluating Alternatives
The second philosophical statement presents evaluation as the process of delineating, obtaining and providing useful information for the purpose of selecting among alternatives. It doesn’t matter if the program was efficiently conducted, effective or impactful. The value of the evaluation is in its being able to compare activities, programs or employees so that decisions can be made in the presence of empirically collected data.

Diversity search committees perform this kind of evaluation. They describe job candidates’ strengths, outline previous experiences, and acquire other useful information that makes it possible to choose among a number of candidates. A company planning to adopt and purchase a computer system will perform this kind of evaluation on all the systems it is considering. It will select the one that performs the best given the needs and resources of the company.

The third way of defining evaluation: Evaluation is the identification of discrepancies between where a program is currently and where it would like to be. For example, an organization’s multicultural marketing department may be to have least one face-to-face focus group with emerging market customers per year. Currently, its sales force sees fewer than half of the multicultural customers in a year. Records of face-to-face calls indicate the discrepancy between where XYZ Corporation is currently as opposed to where the organization wants to be.

Personnel evaluations often take on this definition as well. A new employee’s first evaluation may be an example of the first definition — that is, an evaluation against some minimal standard of performance. After this initial evaluation, certain performance goals are set for the employee. The next and all subsequent evaluations are compared with those performance goals or standards. The discrepancies are identified and remediation strategies are developed.

Chain of Impact
Other levels of evaluation as defined by the Hubbard Diversity ROI Methodology refer to the eventual use of the evaluation data and who might make use of the results

Diversity Return on Expectations, for example, must be based on evidence and impact results. I have found it useful to first to distinguish the “evidence-based, outcome-focused” measures from other types of “activity only” measures. Anyone responsible for implementing diversity initiatives is also responsible for evaluation.

Whether you calculate the impact, from management’s and/or the stakeholder’s point of view, “you will always own the ROI of the initiatives you implement.” So, the amount of evaluation that you provide depends on the types of decisions your organization must make and the information needed to make those decisions.

There are seven levels you can use in the Hubbard Diversity Return-on-Investment, or DROI, evaluation methodology to effectively demonstrate your ROI impact and show a “chain of impact” to meet customer and stakeholder expectations that is evidence-based and credible:

• Level 0: Business and Performer Needs Analysis
• Level 1: Reaction, Satisfaction, and Planned Actions
• Level 2: Learning
• Level 3: Application and Behavioral Transfer
• Level 4: Business Impact
• Level 5: Diversity Return-on-Investment, Benefit-to-Cost Ratio
• Level 6: Intangibles

For example, if your only requirement is to ensure that participants have positive attitudes toward the initiative, then a Level 1 evaluation is sufficient. But, if your goal is to determine whether your diversity initiative is having a positive effect on job performance, then you will have to perform a Level 3 evaluation. This means you will also have to conduct Level 1 and 2 evaluations in order to assess the learning performance applications and job impact at Level 3. They provide the basis for determining whether participants demonstrated that they learned by putting these skills and attitudes to use (verified by a Level 3 evaluation).

There’s no doubt that we must communicate effectively and demonstrate our value to the bottom line. Diversity ROI metrics and performance improvement processes help us focus first on tangible outcomes, then on interventions to meet expectations. When diversity practitioners focus primarily on the intervention — diversity content, the method, or the technology — it’s easy to be led astray by current fads, thus wasting valuable time and money. Instead, focus first on the desired outcomes and DROI analytics to determine what kind of measurable diversity intervention is necessary to meet customer and key stakeholder expectations.

Conducting glitzy diversity training or other diversity activities and implementing fad-based interventions can distract decision makers from what truly counts. The glitz may make things fun, louder and interactive, not necessarily better. Without a clear, data-based front-end analysis of organizational performance gaps, any intervention, including diversity training and the like, is a guess. Add in sophisticated diversity intervention technologies without an adequate front-end analysis, including metrics, and it becomes an expensive and often complex guess.

Systematic diversity training design procedures, for example, must include DROI analytics and metrics, needs assessments, objectives, targeted competency-based design, and multi-level evaluation processes. That framework provides a method to get the coveted seat at the C-Suite table. The DROI metrics and processes you use are solidly based on the behavioral science research results, which provide strategies detailing how diverse people interact and what drives their behavior to produce successful organizational outcomes.

If we examine any other professional discipline or field of study, like medicine, engineering, accountancy, and science, we expect that they are able to prove the efficiency, effectiveness and impact of the solutions, programs and initiatives they what us to support. Why should diversity and inclusion be any different if we want our processes to have credibility and support?