Predicting Change Risk with Reliability

There are several categories of change risk factors and numerous ways to measure them as illustrated in table below.

These can be drawn from many different tools that are used to develop, test, deploy, and support changes, as shown in figure below:

Figure: Sources of Change Risk Factor Data

A robust data engineering capability is required to extract data from diverse data sources such as those mentioned in Figure 2, integrate it, and produce dozens of change risk factor indicators from the integrated data. This provides the inputs to the Machine Learning.

A common question asked by organizations is whether they need to integrate all these different sources and types of data before they can get started. Fortunately, the answer is that you can successfully implement and drive significant benefit using just data from the IT Service Management system, which is why it is starred in figure above. For example, a large global Consumer Products company successfully implemented change failure prediction and cut their failure rate in half just using data from their Service Management system. Over time, you can layer in data from additional sources to provide deeper root-cause insights and enhance prediction accuracy.

Sifting through all risk-factor data outlined in Figure 2, in order to identify patterns that differentiate unsuccessful changes from successful ones would be hard to accomplish manually, even with the best Business Intelligence tools. Machine Learning automates this process of revealing the critical Risk-Factor insights.

Change failure prediction falls in the class of Machine Learning problems called “binary classification.” This means that the Machine Learning algorithm is learning how to classify changes into binary classes: successful versus unsuccessful. As discussed in previous chapters in this series, a variety of binary classification Machine Learning techniques are available, including Logistic Regression0, Random Forests, Gradient Boosted Trees, and Support Vector Machines. It is important to experiment with different algorithms to determine which one works best for a given implementation.

Identifying the optimal solution for a given situation requires approaches such as cross-validation and the use of a hold-out test set. Both of these strategies are based on the notion that randomly selecting a subset of historical changes which were not included in the model's training to serve as test cases.

A frequently asked question is whether poor data quality will result in “garbage in – garbage out”. The good news is that Machine Learning algorithms will identify inputs that have little or no correlation with change failure (perhaps owing to poor data quality) and will avoid them. Hence, they are able to sift through the good and bad data and extract the good.

It is critical to repeat this process of training the Machine Learning model on a frequent basis, as risk factors are always evolving. Risk factors evolve because digital organizations are continually evolving their processes, tooling and resource models among other reasons. Organizations will also take action to mitigate risk factors as they use the AI Change Risk Prediction solution which will result in those risk factors being less significant. The timing of when to re-train the model will depend on when the monitoring of the model’s accuracy in production shows that it is becoming less accurate than when originally trained and deployed.

As an example of a risk-factor insight revealed by the Machine Learning solution, a leading Financial Services company that had extensively modernized its applications and adopted DevOps practices discovered that its remaining legacy applications were experiencing failure rates over 10%. With this insight they were able to focus attention to reduce failure rates to under 4% among these formerly overlooked applications, which resulted in avoidance of annual incident-resolution and change-rework costs totaling $1.4M.

The greatest benefits of change failure prevention will be achieved by taking action on two fronts:

• Removing systemic causes of change failure
• Mitigating risks of individual changes

To eliminate systemic causes of change failure, the machine learning solution must provide insights into how the top risk factors affect change failure rates, allowing the organization to make targeted improvements to people, processes, and technology practices to holistically mitigate these risk factors. This way, the organization is not left waiting for the Machine Learning solution to alert them that changes that are about to be implemented are high risk. Instead, they are taking action holistically to remediate the cross-cutting root causes of change failure. For instance, if machine learning informs you that newly introduced configuration items have a greater failure rate, you may utilize this information to increase quality assurance and testing requirements for updates to newer configuration items.

When the Machine Learning solution flags new changes as high risk, it is vital that the change owners and governance receive particular insights into the risk factors present so they can take appropriate mitigation action. For instance, if the Machine Learning solution flags a new change as having a high failure likelihood due to the change owner's historical failure history, you can take actions such as evaluating the testing and implementation plan or reassigning oversight to a more experienced change owner.

For example, a leading global healthcare company discovered from Change Risk Prediction that Assignee Prior Failure Rate was a top risk factor for new changes failing. The prior track record of the individuals assigned to implement changes had a big impact on the likelihood of changes succeeding or failing. Now, when the Machine Learning solution flags a new change as high risk because the Assignee has a high prior failure rate, they take actions such as reviewing the implementation plan and assigning a more experienced change implementer to provide support. As a result of this and other insights provided by the Machine Learning solution, this organization expects to reduce their change failures by almost 50%.

The figure below illustrates how these two components of change risk mitigation and change failure prevention work together, both powered by the insights provided by the Machine Learning solution.

Figure: Turning insights into action

The dashboard depicted in the figure below enables change owners and change governance to view which changes in the queue are highlighted as having a higher probability of failing, as well as the specific risk factor values causing the high-risk forecast. Additionally, predictions and associated risk factors can be integrated directly into the change workflow via an API interface to the Change Management or Continuous Deployment system.

Figure: Example Change Failure Prediction dashboard