Experimenting with GPT3 to scale quality engineering

Quality engineering has shifted rapidly from person to process, tester to script, automation to intelligent automation, and manual feedback to self-learning scripts in order to address the scale and time to market challenges associated with solutions in today's fast-paced environment.

Figure: Diagram 1

Let’s explore all above topics through real example and understand the benefits. We start with Develop and Deploy phases and will conclude after Monitor & Recalibrate phase.

Quality engineers devote considerable time to identifying the appropriate set of objects and writing a large number of test scripts and test cases. Our case involves a regionally prominent bank that was developing a loan origination and disbursement process. The business desired to develop an application that would automatically generate scores for leads on a daily basis, as customer information was constantly being updated and the criteria for flagging a lead as purse or not-purse could change as additional details became available, altering the probability of sanctioning the loan. Changing data points and features necessitates changing test cases and test scripts, which created havoc for the team. Artificial intelligence techniques enabled us to generate test scripts autonomously based on both baseline and newly added features. How does it work?

1. Identify key topics of use cases, requirement documents
2. Analyze application code using advanced Deep learning-based techniques
3. Generate test cases for data pipelines and modules
4. Generate test scripts based upon acceptance criterion and KPIs
5. Predict foreseen bugs and severity and prescribe the suitable solution?

GPT3 with 175 billion parameters was the backbone of all these experiments. Test cases were framed quite good per requirements and application code.
Every day new data brings another challenge of data ingestion, integrating with other pieces of the puzzle to get business insights in time. To handle this challenge, we used CI/CD pipelines for Data Ingestion which covers:

1. Build
2. Test
3. Release
4. Deploy
5. Validation and Compliance

CI/CD pipeline is a group of steps that must be performed in order to deliver a new version of software.
After CI/CD pipelines, we do not need to wait for final ingestion module to Integrate with other part of the project, so we have all the time latest data and code integrated.

Figure: Diagram 2

Now, through the use of AI algorithms, we have test scripts and test cases scheduled to be accepted as new data arrives.
An AI algorithm analyzed test cases and test logs to identify functions that were highly malicious, along with a probability score, in order to take appropriate action and automate even self-healing techniques in the future. AI-based program enables round-the-clock operation of lead scoring applications, while testers can execute tests as and when required with no reliance on application execution. Above all, it occurs in real time, which improves precision and accuracy.
For this use case, AI algorithms have been integrated into the various stages of the software lifecycle.
Self-learning and self-healing pipelines enabled models to become more intelligent with each consumption and decision.
At the lead generation stage, due to the large number of data sources – structured and unstructured – it is not possible to ingest and feed everything to determine whether or not intelligence decision making is occurring, and it is also a lengthy, expensive, and time-consuming exercise. By analyzing historical project data and QA costs, AI systems were able to forecast the cost of quality across the software lifecycle and the priority of each feature.
AI was extended to predict which modules or components in the software code are likely to have more defects, assisting in the definition of more effective testing strategies, automating test design, and increasing test coverage. AI analyzed production data to determine which procedures and functions are frequently used. We predicted what changes might be required in specific modules based on this insight, thereby reducing the time and effort required to design and execute regression tests, which could have taken considerable time in rolling out the solution for final decision making.
By mining patterns from the previous release's defect, root cause, severity, and discovery phase, we identified the most likely defect for the next release and automated the optimization of relevant test cases and test scripts.

We saw firsthand how we achieved QE at scale during development and design by implementing AI techniques. Once the code was deployed to production, another significant challenge awaited us. How frequently should we monitor our solution to ensure that it continues to deliver the expected results in accordance with business objectives? When we need to update our solutions with new data, features, or techniques, or even when we need to decide when to rebuild and re-tire.

All of these decisions are certain to require considerable effort and create a reliance on experts. Once multiple solutions are implemented across multiple business processes, multiple countries, and regions, this model becomes unscalable. What is the solution, then? Consider our case where we deploy to production and then use AI to automate the above actions.
Refer to the following architecture diagram to see how we enabled auto-monitoring, auto-recalibration, and approval-based training of production solutions.

Figure: Diagram 3

The architecture described above is hosted on the AWS cloud. The first step is to infer the deployed solution. It published predicted results and compared them to ground truth (real-world data) in order to determine accuracy. If accuracy degrades, it may be necessary to re-train or re-calibrate the solution, but obtaining ground truth data requires its own process time and cannot be accomplished prior to the actual date/time, posing a clear risk of solution depreciation and failure to meet expected KPIs. To overcome this, we used statistical techniques to analyze newly arrived data sets. Several techniques have been used, including KS Statistics, Population Stability Index, Z-proportion test, and Wasserstein distance techniques, to alert us to significant changes in the distributions of input data for influential features that are otherwise unchanged. All of these tasks have been automated and scheduled using DevOps pipelines.

Another difficulty we encountered was managing multiple versions of the same solution in production, with the requirement of dynamically redirecting inputs to the correct version of the solution. We used A/B testing on the AWS platform to validate the accuracy of each model on a specific dataset and then auto-redirected data to the best performing model to generate results for final decision making, as shown below:

Figure: Diagram 4