Chapter 4 by Capgemini
Business ●○○○○
Technical ●●●●○
Listen to the audio version
We all witnessed transformation from QA in waterfall SDLC to Quality Engineering to ensure quality is built into each phase of development instead of checking at the end. Further, to industrialize QE at Scale, we adopted predictive and prescriptive techniques using AI & Machine Learning advanced algorithms.
Many of us have executed at least double-digit waterfall programs till now and first thing is open secret that it takes months and years. Around 20-25 years back, we had large engagements and builds were performed only few times in a year, so we have enough room to perform quality assurance at the end of development. We wrote test cases for months and quarters and achieved successful KPIs but today, in 2021, builds are happening multiple times within a month or even within a day. We also leveraged sophisticated tools and programming, but overall process remains the same. Thus, wearing forward-looking glasses, it is clear, we need to bank up Intelligent automation @scale at every step of quality engineering, it should move from waterfall to real-time Agile, adopt a theme Be like Water – Free from any Shape or Form but still Agile, Adaptive and Resilient. Next question comes in our mind. How? Answer lies in powerful AI & Machine learning algorithms which not only automate steps intelligently but also predict and prescribe modules, phases, features which are more error-prone and provide self-learning and self-learning techniques to avoid human manual intervention and avoid bias in quality engineering process.
Let’s understand further through real life examples:
This chapter discusses the real production grade implementation of Quality Engineering (QE) at Scale using Artificial Intelligence for of the leading financial institutions in the globe.
Quality engineering has shifted rapidly from person to process, tester to script, automation to intelligent automation and manual feedback to self-learning scripts to achieve scale and time to market challenges of solutions due to fast paced competition.
Figure: Diagram 1
Let’s explore all above topics through real example and understand the benefits. We start with Develop and Deploy phases and will conclude after Monitor & Recalibrate phase.
Quality engineers spend lot of time in identifying the right set of objects, write tons of test scripts and test cases. Our case is related leading bank in the region which was working on developing a Loan Origination and disbursal process. Business was looking to develop an application to generate score for leads on daily basis because customer information was getting enhance on daily basis and flagging lead as purse or Not-purse also may change with additional details and result in change the probability of sanctioning the loan. Changing data points and features results in changing test cases and test scripts and it was nightmare for the team. AI based techniques helped us to generate test scripts autonomously based upon baseline as well as new added features. How does it work?
GPT3 with 175 billion parameters was the backbone of all these experiments. Test cases were framed quite good per requirements and application code.
Every day new data brings another challenge of data ingestion, integrating with other pieces of the puzzle to get business insights in time. To handle this challenge, we used CI/CD pipelines for Data Ingestion which covers:
CI/CD pipeline is a group of steps that must be performed in order to deliver a new version of software.
After CI/CD pipelines, we do not need to wait for final ingestion module to Integrate with other part of the project, so we have all the time latest data and latest piece of code integrated.
Figure: Diagram 2
Now, using AI algorithms, we already have test script and test cases which have been scheduled to get excepted once new data arrives.
Test cases and test logs have been analyzed by an AI algorithm to investigate functions that were highly malicious along with probability score to take proper action and automate even self-healing techniques going forward. AI based program enabled to run lead scoring application around the clock and tester can execute tests as and when required with zero dependency upon application execution. Above all, it happens in real-time which enhance the accuracy and precision.
AI algorithms have been integrated within the different stages of the software lifecycle for the use case.
Self-learning and self-healing pipeline using DevOps, made sure that models are getting becoming intelligent with every consumption and decision making
At Stage of lead generation, due to large number of data sources – structured and unstructured, it is not possible to ingest everything and feed everything to check if Intelligence decision making is happening or not and moreover, it is lengthy, costly and time-consuming exercise. Here, AI systems have predicted the cost of quality across the software lifecycle and priority of each feature by taking into account historical project data and QA costs.
AI extended its assistance to predict which modules or components in the software code can be expected to have more defects and helped to define better testing strategies, automate test design, and improve test coverage. AI leveraged production data to analyze the procedures and functions and determine which ones are extensively used. Based on this insight, for new requirements, we predicted what changes might be needed in specific modules, reducing the regression test design and execution time and effort which could have taken considerable time in rolling out solution for final decision making.
Using patterns mining of last release defect, root cause, severity, discovery phase, etc., we identified the probable defect for next release and optimized relevant test cases and test scripts automated.
We have witnessed how we achieved QE at scale during develop and design using AI techniques in practice. Now, once code has been deployed into production then another biggest challenge was waiting for us. How frequently we should monitor our solution so that it can continue to provide expected results per business objectives. When we need to refresh our solutions with new data, new features, new techniques or even decide when to rebuild and re-tire.
All these decisions are bound to take lot of effort and create dependency upon experts. Once we start implementing multiple solutions across multiple business process, multiple countries and regions, this model cannot be scalable. What is the solution then? Let’s see our case when we deployed into production then how we to automate the above actions using AI.
Let’s refer below architecture diagram and understand how we have enabled auto-monitoring and auto-recalibration and approval-based training of solutions in production.
Figure: Diagram 3
The above architecture is at AWS cloud. Step 1 proceeds with inference for deployed solution. It published predicted results and matched with ground truth (real data) to calculate accuracy. If accuracy dips then it can one of the reason of re-training or re-calibration of solution but arriving ground truth data takes own process time and it cannot be possible before actual date/time and we have clear risk of solution depreciation and missing expected KPIs. To overcome it, we adopted statistical based techniques to analyze new data sets once it arrives. Few techniques which we have used are KS Statistics, Population Stability Index, Z-proportion test and Wasserstein distance techniques to notify if we see significant change in input data distributions for influential features otherwise status-quo. All these tasks have been developed through DevOps pipelines to automate and schedule it.
Another challenge, we have faced to manage multiple versions of same solution in production where requirement is to re-direct inputs dynamically through right version of the solution. We implemented A/B testing at AWS platform to check accuracy of each model on particular dataset and then auto re-direct data through best performing model to produce results for final decision making which is reflected as below:
Figure: Diagram 4
Jatinder Kumar Kautish
Jatinder K Kautish is a Director at Artificial Intelligence Offering, L3 Certified Chief Architect, IAF Certified, working from India in Hyderabad. He is a regular Industry speaker at leading conferences and academic institutes. He has been awarded with 3 AI Innovation Awards in 2020-21 and performing an Advisory panel for Confederation of Indian Industry (Govt. of India) for 2020-21. He has a passion for positing AI technology at the core of every business and converting solutions into established offerings. Outside of work, you’ll likely find him mentoring academic and NGO technology projects or writing & reciting poems and training ambitious folks for Punjabi folk dance (Bhangra) or enjoying long drives.
Capgemini is a global leader in partnering with companies to transform and manage their business by harnessing the power of technology. The Group is guided everyday by its purpose of unleashing human energy through technology for an inclusive and sustainable future. It is a responsible and diverse organisation of 325,000 team members in nearly 50 countries. With its strong 55 year heritage and deep industry expertise, Capgemini is trusted by its clients to address the entire breadth of their business needs, from strategy and design to operations, fueled by the fast evolving and innovative world of cloud, data, AI, connectivity, software, digital engineering and platforms. The Group reported in 2021 global revenues of €18 billion.
Get the Future You Want I www.capgemini.com
