Quality is never an accident. It is always the result of intelligent effort.
(John Ruskin)
The use of analytics and ML-based dashboards assist us in making informed decisions and increasing the effectiveness and efficiency of our activities.
In the previous chapter, we examined the importance of quality gates in our culture of continuous improvement. They contribute to increasing the level of transparency and improving our processes. More significantly, they rely on metrics to provide insight into our team's test progress, productivity, and the quality of the system under test. However, while our development techniques have evolved significantly over the previous five years, our metrics have mostly remained consistent.
Questions such as "what to test," "what to automate," "how to automate more," and "when to stop testing while maintaining the same level of quality" are still mostly answered by educated estimates and expert judgments.
What if we could address questions such as “Was our testing on target?”, “Why did issues happen?”, “How much time or effort will it take to fix future defects?”, “What should we test now?”. There are so many possibilities for establishing metrics in testing that the key criterion both teams and organizations as a whole should set for themselves is to ask:
In this chapter, we review the fundamentals of valuable metrics, and how AI can provide us with even smarter QE analytics.
To set the ground, let’s spend one moment on various approaches to identifying metrics. TMAP recommends to approach metrics through an iterative and collective process. They define good metrics as being business centric, improvable, and inspiring action. metrics should be used for the right reasons. For instance, any indicators of test automation should keep us focused on the overall quality, rather than on achieving 100% automation. Computed metrics should allow us to:
Balancing effectiveness and efficiency
Indicators help us monitor our ongoing activities and identify areas for improvement. As such, we must constantly review the effectiveness and efficiency of our efforts. Efficiency and effectiveness are two closely related notions with a subtle but nonetheless significant semantic difference.
Figure: Productivity quadrant
The value of metrics
In 2019, Tricentis commissioned Forrester to conduct a research on what quality metrics are most critical for DevOps success. The findings have been compiled into a 55-page ebook, which categorized all the metrics across 4 value sections.
Figure: Metrics quadrant
The 2020 State of Continuous Testing report shared that the top three metrics for the effectiveness of continuous testing focused on business results (production data, user feedback, and ROI). More than three-quarters of respondents (78%) said that “getting visibility throughout the development lifecycle” is a challenge when implementing continuous testing. This shows that organizations are maturing, and that they are looking more closely at the delivery of overall quality and value to the business and the end-customer.
Figure: Survey question from the 2020 Continuous Testing Report
When asked how they were tracking the effectiveness of continuous testing, the top three responses focused on the outcome of the functionality (production data, user feedback, and ROI). This suggests that the majority of respondents are more concerned with business outcomes than with process-level KPIs. However, because KPIs at the process/technical level are more actionable, teams will want to measure the metrics that support their work and decision making. As continuous testing matures, lower-level KPIs such as level of automation, requirements coverage, and defects found improve, and those results are seen in the top- level business metrics that reveal overall quality and value to the business and the end customer.
Quality metrics against the release frequency
For many, continuous delivery is the desired end state, allowing new functionality to be released as soon as it is developed. We believe the choice in quality metrics for a given product is highly dependent on this release cycle, be it immediate and incremental, a month long, or even longer. In the below table, we propose a mapping between the delivery schedule and examples of metrics.
| Less than one day | Between one day and one week | Between one week and one month |
|---|---|---|
|
Same measurements as of “less than one day- deployment frequency”, in addition to that,
|
Same measurements as earlier, in addition to Change based automated selection & execution:
|
We have just examined that data can easily turned into metrics to monitor and track activities. Metrics mostly focus on the ‘what’, to gauge the performance or progress of our activities. However, metrics alone will not assist us in taking action, comprehending what is occurring, or improving results. Quality engineering activities generate a large volume of both structured and unstructured data that is frequently left idle and unused.
Recent development in AI can help turn data and metrics into smart analytics to assist us in making better decisions. For instance, it may render some of our prior tests obsolete, if comparable data is available. Additionally, we may be able to detect low-priority or low-value tests, such as those for features that are not being used.
Analytics can be collected to help validate our quality-in-use criteria and to get feedback on the effectiveness of the tests performed and those that were omitted for that release. Ideally, there will be no reported crashes or faults if our testing was "purpose-fit" (assuming any related issues were fixed prior to release). Additionally, as we acquire knowledge from analytics, we can then develop more effective testing for future application releases.
Similarly, product development manager will get insight into code churn happened in each build, build quality and test success rates in every phase of testing. Analytics can also serve to predict the percentage of risk of finding critical defects in production.
Figure: example of predictive analytics
For the purpose of this chapter, we will focus on 3 categories of analytics capabilities, as described below:
| Descriptive analytics | Predictive analytics | Prescriptive analytics | |
|---|---|---|---|
| Objective | Descriptive analytics can make testing data more informative and ascertain what occurred in the prior test cycles. That appears to be standard orchestration intelligence, but the source data spans across multiple repositories. | Predictive analytics allows us to comprehend whatever insights we now possess and forecast what may occur. Machine learning can produce valuable insight into resource and time allocation. | Prescriptive analytics help translate the above observations and forecast into tangible actions and generate concrete recommendations for what should occur, including the implementation of those recommendations. |
| Focus | Report and analysis based on historical data | Forecast and simulation to foresee critical issues, reduce reaction times, automate resolution | Alerts and recommendations to optimize testing and increase defect detection |
| ML-driven use cases |
|
|
|
Figure: Additional examples of smart analytics
Concrete example using Cognitive QA
Cognitive QA is a real-time analytics platform developed by Capgemini that allows QA professionals to improve decision-making. While certain indicators are available but dispersed across multiple platforms, the goal is to function as a single control tower. Cognitive QA helps address typical challenges such as the inability to complete regression test cycle due to the volume of test cases, limited time, resource constraints or the lack of test automation strategy.
Figure: high-level architecture of Cognitive QA
As the lack of data readiness and asset mapping equate to deployment ineligibility, an assessment of the quality, volume and integrity of data is performed before any consideration. Further investigation is performed around the following aspects:
| Descriptive analytics | Predictive analytics | Prescriptive analytics |
|---|---|---|
|
|
|
For instance, Cognitive QA enables us to do in-depth analyses to ascertain both the exact causes of problems and their impact on the system. In a typical manual approach, root cause analysis (RCA) can take days to weeks to complete. The ‘root cause analysis report’ illustrates an automated RCA report. It allows to take deep dive by project, application, module with automated pareto distribution. This really aids in determining where further resources/time should be invested to resolve challenges. This report employs a quasi-Monte Carlo algorithm with ranking, inclusive of a few more NLP techniques. One can employ a time machine to rerun the analysis. This enables us to draw lessons from previous releases and fine-tune our strategy.
In the “what to test” figure, we examine how to increase quality with risk-based testing for every development build. Cognitive QA assists us in making decisions about which test cases to execute for a specific application or end-to-end flow. This can be accomplished through the use of a natural language processing (NLP) ranking approach.
Often, the time available for the test cycle is so restricted that we are unable to finish the full regression cycle. However, we are under pressure to create high-quality work. Additionally, we may add a timing dimension to the data and prescribe a fixed set of test cases to ensure the highest possible test coverage in the allotted time. The Cognitive QA platform can help us identify high-priority test cases within a project by assigning a rank to each test case. Test selection ranking is determined using the test selection score, which is calculated by combining the execution score, the failure score, the defect score, the recency score, and the risk score for each test.
Example 1:
A storage corporation in the United States was examining how to prioritize tests and configurations depending on the likelihood of risk and failure. The idea was to save time and make better use of available hardware.
Example 2:
A vendor of medical imaging solutions was considering modifying their technique for selecting regression tests based on different inputs. The recommended test list should be included into the cycles of automated test executions. The platform correctly suggested focusing on 1,664 regression test cases from the 13, 721 existent ones.
We must also consider the frequency with which test cases were conducted previously, the velocity of code changes, the interdependence of applications, test cases, and test data, as well as compliance, security, performance, and return on investment requirements. The figure "what to automate" assists us in prioritizing automated test cases. This can be performed by utilizing a natural language processing (NLP) ranking solution.
Having little to no visibility of risks and faults leakage in production is a recurring challenge, with direct impact on the business. For years, numerous methods have been employed to forecast problems with the high level of accuracy. The challenge is that no forecasting model can be simply estimated statistically from defect trends. It must also take into account code check-ins, requirement volatility, and a variety of other factors. The ‘defect prediction’ figure illustrates the range of the prediction model's output. Algorithms such as linear regression allow to reach 98% of accuracy.
Now we have arrived at the critical choice point of "to be or not to be." The one decision that can propel a business to the next level or can erode customer trust and have a negative impact on revenue and brand reputation. There is always debate over whether we test enough or we test too much, and eventually, how we can deliver products on time and with the highest quality. As the DevOps pipeline process matures, it's also critical to consider how quickly we can respond to errors that occur in production. Occasionally, we can take a gold build, stabilize it in a test environment, and ensure that all problems discovered are addressed through hotfixes. We may also immediately release products and hotfixes. When to cease testing gives you all the information you need to make the best choice. It gives you with all the information necessary to make an informed decision. It generates recommendations based on machine learning, natural language processing, and key KPI inputs.
Maheshwar Kanitkar
Maheshwar is a Senior technology executive with over 25 years of rich experience in large corporate and start-up environments. During this period, Maheshwar has performed several leadership roles in driving business (top-line and bottom-line), Innovation, R&D, Product Management, Partnership, Analyst relationship, Workforce Productivity, Talent Transformation, Project delivery. Maheshwar's Strength is in driving transformation internally within the organization resulting in significant growth and bottom-line impact and also working closely with the customer in helping them achieve their business transform. Maheshwar lead Innovation COE for AI and QE. Maheshwar has Hands-on expertise in Cloud, AI, Automation, Dev-OPS, test automation technologies, building High-Performance Multi-Cultural Leadership Team around the globe, and turn around project delivery. Maheshwar leads testing portfolio sales & solutions for Capgemini Europe.
Vivek Jaykrishnan
Vivek Jaykrishnan is an enterprise test consultant and architect with extensive experience. He has over 22 years of experience leading Verification and Validation functions, including functional, system, integration, and performance testing, in leadership positions with reputable organizations. Vivek has demonstrated success working across a variety of engagement models, including outsourced product verification and validation in service organizations, independent product verification and validation in captive units, and globally distributed development in a product company. Additionally, Vivek has extensive experience developing and implementing test strategies and driving testing in accordance with a variety of development methodologies, including continuous delivery, agile, iterative development, and the waterfall model. Vivek is passionate about incorporating cognitive intelligence into testing and is also interested in exploring the frontiers of IoT testing.
Antoine Aymer (chief editor)
Antoine Aymer is a passionate technologist with a structured passion for innovation. He is currently the Chief Technology Officer for Sogeti's quality engineering business. Antoine is accountable for bringing solutions and services to the global market, which includes analyzing market trends, evaluating innovation, defining the scope of services and tools, and advising customers and delivery teams. Apart from numerous industry reports, such as the Continuous Testing Reports, the 2020 state of Performance Engineering, Antoine co-authored the "Mobile Analytics Playbook," which aims to assist practitioners in improving the quality, velocity, and efficiency of their mobile applications through the integration of analytics and testing.
Part of the Capgemini Group, Sogeti operates in more than 100 locations globally. Working closely with clients and partners to take full advantage of the opportunities of technology, Sogeti combines agility and speed of implementation to tailor innovative future-focused solutions in Digital Assurance and Testing, Cloud and Cybersecurity, all fueled by AI and automation. With its hands-on ‘value in the making’ approach and passion for technology, Sogeti helps organizations implement their digital journeys at speed.
Visit us at www.sogeti.com
Capgemini is a global leader in partnering with companies to transform and manage their business by harnessing the power of technology. The Group is guided everyday by its purpose of unleashing human energy through technology for an inclusive and sustainable future. It is a responsible and diverse organization of 325,000 team members in nearly 50 countries. With its strong 55 year heritage and deep industry expertise, Capgemini is trusted by its clients to address the entire breadth of their business needs, from strategy and design to operations, fueled by the fast evolving and innovative world of cloud, data, AI, connectivity, software, digital engineering and platforms. The Group reported in 2021 global revenues of €18 billion.
Get the Future You Want | www.capgemini.com