Predicting Success

Patti P. Phillips, Ph.D., Chief Executive Officer ROI Institute, Inc.
Jack J. Phillips, Ph.D., Chairman, ROI Institute, Inc. 

Part of the evaluation mix should be the predictions of success. Predictions serve as leading indicators of success and facilitate process improvement. Think about the logical chain of value from any learning program: Level 1, reaction; Level 2, learning, Level 3, Application; Level 4, impact; and ROI, Level 5. We know from experience that programs are not evaluated very often at Level 3, less often at Level 4, and rarely at Level 5. Unfortunately, these are data sets that executives would like to see. Can we use the lower levels to predict the higher levels? The answer is yes, but first, there are four issues that should be explored when prediction is considered.

1. Precondition versus prediction. Sometimes, one level is a precondition to the other, but not necessarily a predictor. For example, learning is a precondition for application, but just because a participant learned doesn’t mean that there will be application.
2. Barriers and enablers. For any program to achieve success, there are often barriers, which get in the way of actually using the learning. Enablers help participants learn and apply that learning. Barriers and enablers can inhibit or distort predictions.
3. Learning should produce a business impact. To supporters and sponsors, success is not achieved when there is learning. There must be application and then impact.
4. Beginning with the end in mind. Many programs don’t begin with the end in mind, an impact measure. For most sponsors, the end of a program should be the impact it has on the organization, individual, or community.

 Can Level 1 Predict Level 3?

Reaction will predict application. Individuals in a program will usually decide to use the content (or not) based on their reaction to what they are learning. (i.e., Learning influences reaction.) Reaction drives application and this becomes a predictor, but not every reaction will cause application.

Based on our research (as well as the research of our clients), here are the predictors, listed in the order of strength of correlation between reaction and application:

1. Will you use the knowledge and skills in your work?
2. Will you recommend the program to others?
3. Is this program important to your success?
4. Is this program relevant to your work?

These are usually good predictors of success because they are content-related measures, not experience. It is the content that will make the difference later, not necessarily the experience.

Can Level 2 Predict Level 3?

Learning can influence reaction if the content is important to participants, relevant to their work, and something they will use or recommend. This reaction, which comes from learning, can be a predictor. The learning itself is not a good predictor because of the barriers that often inhibit the transfer of learning. A significant correlation between test scores and application may not be common.

 Can Level 3 Predict Level 4?

Application will normally predict impact, if the program started with the impact and the learning is the appropriate solution to improve the impact measure.

Can Level 1 Predict Levels 4 and 5?  

Reaction can predict impact if the program starts with the end in mind (impact) and participants and stakeholders clearly envision the end to be at Level 4. If it can predict Level 4, it can predict Level 5, the ROI, because the ROI calculation is based on improvements in the actual impact.

In some programs, participant feedback at Level 1 includes a prediction of application and impact, and this can be used to create a forecasted ROI. This becomes very powerful, particularly when there is some concern about the program delivering business value.

Preprogram ROI Forecast

An increasing number of executives and sponsors are requiring a forecast of impact or ROI before the program is implemented. Given tight budgets and scarce resources, this is a reasonable request. For most programs, this can be achieved with minimal effort by experts who know the content of the proposed program and experts who understand the context (the role of participants, the environment where they work, and the impact measure they are influencing). Application and impact objectives are created by these experts. Estimates of impact are developed and error adjustments are made to produce credible forecasts. 

 There you have it. You can predict success using learning analytics. The CLO must use predictions and forecasts, particularly when the resources are not available to routinely measure success at Levels 3, 4, and 5. For examples of these predictions, correlations, and forecasts, please email [email protected].