State of AI applied to Quality Engineering 2021-22
Section 3.1: Inform & Measure

Chapter 1 by Sogeti

Smart quality gates

Business ●●●○○
Technical ●●○○○

Listen to the audio version

Download the "Section 3.1: Inform & Measure" as a PDF

Use the site navigation to visit other sections and download further PDF content

By submitting this form, I understand that my data will be processed by Sogeti as described in the Privacy Policy.*

The delivery of IT solutions with a quality focus requires intelligence, which can be augmented by the use of smart quality gates. Such gates can be gradually deployed in accordance with the maturity DevOps journey.

We all know that digital business requires both efficiency and a quality-driven mindset. While Jez Humble's CALMS (Culture, Automation, Lean, Measurement and Sharing) framework[1] can help us pave the way to more efficient delivery, the pursuit of quality necessitates a continuous testing and quality assurance strategy[2]. Both of these principles must be pursued within a context of limited resources, which makes it critical to constantly analyze, reason, prioritize, and choose between alternatives.
As writer John Ruskin put it, “Quality is never an accident. It is always the result of intelligent effort”. Indeed, the term "intelligence" derives directly from the Latin words intelligentia and intelligere, which mean “to understand, comprehend, come to know” and it is the combination of two primary words: inter (between) and legere (choose). In summary, delivering IT solutions takes intelligence to maintain a constant balance of risk and quality when delivering business value[3].

In this chapter, we propose a strategy for developing an intelligence base to support and enrich project quality gates. Naturally, a set of enhanced metrics is also needed to augment our intelligence, which we will examine further in the next chapter.

[1] R. Marselis, B. van Veendendaal, D. Geurts y W. Ruigrok, Quality for DevOps teams, Sogeti, 2020 
[2] Sogeti, «Continuous Testing Report,» 2020
[3] A. Tort, «Is (Artificial) Intelligence Needed for Testing?,» SogetiLabs blog, 2018

Getting started with smart quality gates

The creation and evolution of intelligent workspaces is critical for efficient and quality-driven IT delivery, and it is now possible due to increased computational capabilities (based on the well-known "Moore Law") that allow the application of narrow artificial intelligence to the quality assurance domain for software projects[4]. The 2020 World Quality Report[5] concludes that use cases such as “use predictive test selection and optimizations”, “optimize test schedule and effort in a risk-free manner”, “release risk prediction” or “automated defect remediation”. To improve our decision-making process, we must acquire and apply knowledge. Acquiring knowledge in IT projects entails gathering data, converting it to information, and using it as knowledge to manage the project, as well as a continuous learning process that accumulates knowledge as experience.
Such intelligence can be fragmented and implicitly stored in the brains of human project professionals. Owing to human memory constraints, excessive knowledge contexts, ambiguity, or the need to act fast[6], this often results in intuition, subjective disputes, and cognitive bias that violate the "what you see is what there is" (WYSIATI) requirement. By comparison, we recommend making the intelligence process used in DevOps more explicit as a foundation for augmenting human intelligence with artificial intelligence.
DevOps is typically applied by (as much as possible) automated pipelines that define the explicit steps required to move software from development to production. However, as new software contributions evolve to production through development and deployment activities, the risk of defects increases, as each operation can result in a variety of potential defects. As a result, a special type of DevOps pipeline activity called Quality Gates must be defined, implemented, and orchestrated throughout the pipeline.

[4] Sogeti, «AI First: Learning from the machine,» 2017
[5] Sogeti, Capgemini and Microfocus, «World Quality Report,» 2020
[6] D. Kahneman, Thinking, Fast and Slow, 2011

A clear change in value

Quality gates are defined as a “certification that all appropriate work required to move products forward -a gate- to subsequent project activities has been completed and reviewed, and product meets specific quality expectations”. Our definition of Smart Quality Gates for DevOps contexts is based on [7]: “Significant milestones and decision points with predefined and quality focused criteria”. However, in many contexts, quality gates have been simply conceived as checklists or formal signoffs that indicate completion of a particular software development artifact. Additionally, these type of quality gates were manually informed and controlled, often using binary passing criteria.

Our approach is diametrically opposed to the conventional approach of a late massive validation step prior to product delivery, and is more akin to a continuous series of checklists (as when operating an airplane) to minimize risks and ensure quality consistency throughout the lifecycle.

By adapting this concept to our context, we describe a Smart Quality Gate as a critical decision point with quality-focused parameters which can be tested automatically using project (human and artificial) intelligence. Such a gate aims at mitigating quality risks as software contributions progress across DevOps pipelines.

Taking this definition into account, the table below states the main components of a quality gate

Quality Gates components
Component Description
Entry criteria The subset of project input data that is required to be considered by the quality gate.
Exit criteria metric A (automatically or semiautomatically) computable measure about the current quality of the project. This computable measure may be a simple input, a simple calculation, or an advanced analytics & AI based model execution.
Exit criteria threshold A configurable threshold (decision value), the meaning of which is directly related to risk reduction. The quality gate will be passed if the exit criteria metric value in each moment is higher than this threshold.
Exit criteria measure unit The unit measure in which the threshold is computed.
Severity It defines the type of reaction when a quality gate fails. The default reaction is stopping the pipeline and refuse the software contribution. An alternative is only providing a warning, but not stopping the pipeline.
Notification policy It defines the notification policy for quality gates passes and fails.

[7] T. Flohr, «Defining Suitable Criteria for Quality Gates,» de International Conferences on Software Process and Product Measurement, 2008

Time to implement smart quality gates

Quality Gates must be implemented in the corresponding orchestration job server as pipeline steps. We have deployed quality gates into a variety of platforms, like Azure DevOps, Jenkins, Bamboo, and GitLab. Each quality gate must establish connections to the intelligence system in order to meet its admission requirements.

Additionally, the metric's computation must be applied, and its output must be exposed to the Quality Gates configuration and management layer. The pipeline will either proceed or be shut down, depending on the severity. Similarly, the corresponding notification behavior will be activated in accordance with the notification policy.

The core of Smart Quality Gates approach is the setting up, maintenance and evolution of an intelligence system capable of providing information as a service to Quality Gates implementations for use in the DevOps workspace.

The intelligence framework must collect data explicitly and automatically through connectors from the project's various data repositories (test cases, defects, code, user support, etc.) and structure it as a knowledge base model.

Additionally, advanced analytics and cognitive models can be implemented to provide advanced intelligence features that are exposed to the Quality Gates implemented in DevOps pipelines via Application Programming Interfaces (API). Such Quality Gates can also report back exit criteria that can be used to enrich learning procedures within the intelligence system.

The need for a quality gates manager

What we have observed in many DevOps contexts is that quality gates have two clear sides:

  • The technical implementation within a pipeline based on the connections and computation features of the DevOps platform.
  • The definition, check and control of exit criteria (thresholds) and the management of the reactions based on such criteria execution and results.

We suggest that the management and operation of both sides be explicitly separated, especially because they are assigned to distinct project roles. This is why we advocate for the development of a dedicated layer to handle and monitor quality gates in a non-technical manner, including the configuration of exit requirements thresholds, severity, and notification policy.

The Figure below shows an example implementation made by Sogeti to manage quality gates from a business risks perspective.

Figure. Example screenshot of Quality Gates Configuration & Management Layer

Figure. Example screenshot of Quality Gates Configuration & Management Layer

The quality gates manager may be complemented by the implementation of automatic dashboards that provide the big picture about the quality status and evolution of a project, or the whole set of projects in the organization. The implementation of this layer may be done through Sogeti Dashboards but taking most of the built intelligence system as input source.

A phased deployment based on maturity

We implement the smart quality gates approach in an incremental way, based on three dimensions: (1) a maturity journey of the smart quality gates system, and (2) the progressive addition and enriching of smart gates, and (3) the evolution of the intelligent system.

By this way, DevOps environments may evolve through the following stages regarding smart quality gates system:

  1. Reactive QA in DevOps. No quality gates are explicitly implemented within pipelines, so QA activities are unstructured, reactive and not integrated as critical activities in the IT delivery process.
  2. Integrated QA gates in DevOps. Some Quality Gates are implemented as continuous validation steps in DevOps pipelines, boosted by automated actions, but directly connected to an explicit intelligent system.
  3. Smart and integrated QA gates in DevOps. An intelligence system is set up to act as a smart knowledge provider to enrich the capabilities of QA gates.
  4. Smart, integrated and managed QA gates system in DevOps. In addition to the previous journey level, a non-technical layer for managing QA Gates is added, in conjunction with a dashboarding system.

Smart, integrated and managed QA gates system within DevOps
The figure below shows the general overview of our Smart Quality Gates approach. The main aim is to set-up a working environment to deliver IT value based on:

  1. Providing intelligence systematically.
  2. Orchestrating quality gates implemented in DevOps pipelines.
  3. Providing executive analysis for transparency and management.

This approach is also layered-based with the aim of associating both technical and business professionals. These are the main components of the approach:

  • Data acquisition through connectors from project data repositories that support the operative management of projects.
  • Intelligence system that consolidates, analyzes and centralizes project data and turn it to a knowledge base, with AI models (learning, prediction and advanced analytics) to provide intelligence to DevOps pipelines.
  • Implementation of quality gates as an incremental Quality Assurance strategy to progressively reducing the risks as new software contributions cross each gate.
  • A configuration and management layer for configuring quality gates based on thresholds that reflect a company's quality risks model for business professionals, even with non-technical skills.
  • A global dashboard for quality transparency and measurement that depicts the project's overall quality.
Figure. Smart Quality Gates approach overview

Figure: Smart Quality Gates approach overview

Each analytics and AI-based model requires an standardized API structure to be integrated into the intelligence system, consisting of a set of methods that may be used in the technical implementation of pipeline steps, including the required input values and the expected outputs. Each API method became a usable intelligence service chunk to be used in pipelines to build the quality gates, but it is expected to include the following items:

  • A computational model that includes statistical or estimation methods to provide advanced analysis on source data. The model may also use Natural Language Processing (NLP) techniques, spiders and sentimental analysis for data processing, analysis and transformation.
  • A computable level of confidence about the model results.
  • AI-based techniques (machine learning) for incremental learning based on previous data.

Each intelligence service chunks (descriptive, prescriptive and predictive) constitutes the evolving QA brain for automatically supporting projects and enhancing quality gates to be smarter and integrated to the DevOps workspace.

Use Case: Building a progressive smart QA workspace in the water management sector

Smart QA Gates is introduced incrementally over a three-year period for one of the world's leading environmental service companies in the water management market. Three significant obstacles had to be overcome:

  1. Increase the emphasis on quality assurance and organize it to be more transparent, structured, and assisted
  2. Convert static testing activities into QA gates that help the agile/DevOps transition
  3. Define, measure, and monitor optimization objectives using metrics and key performance indicators (KPIs) supported by advanced analytics and artificial intelligence

To address these issues, a QA maturity plan was developed and accompanied by a phased implementation of smart QA gates:

  • Phase 0. Preparation for robust testing & QA activities.
    The roadmap began with a significant evolution of QA operations, followed by the definition and implementation, as quick wins, of simple KPIs and visual dashboards for executive management to monitor quality. These metrics required the adaptation of specific connectors and analytical algorithms to the diverse context of multiple development suppliers, data sources, and stakeholders.
  • Phase 1. From Reactive QA in DevOps to Integrated QA gates in DevOps.
    The results of phase 0 served as the foundation for the continuing agile & DevOps transition journey. Additionally, a culture of metrics was introduced as a source of knowledge for driving the progression of certain areas and initiatives through the pipelines.
  • Phase 2. Scaling to an intelligent QA system for projects.
    Individual and isolated metrics, dashboards, and information used to drive specific pipelines were scaled up and centralized at the organizational level to create a true intelligence framework. Its main purpose is to add value to operational decisions and facilitate the operation of automated pipelines. In this phase, Four AI-based models were introduced in this process to expand the intelligence system's predictive capabilities (requirements risk, test cases prioritization, defects and test cases duplication checker, and a machine learning model to support the challenge of maintaining the regression test cases up-to-date). Such AI-based models contributed to increase the confidence and the smart capabilities of the solution. Both automated and manual testing and code quality process results were introduced as data sources for various technologies and platforms (web, SAP, mobile,...), while the intelligence system also fed project pipelines for various environments.
  • Phase 3. Towards a Smart, integrated and managed QA gates system.
    As the number of pipelines, templates, and dashboards grows in the enterprise, the current phase focuses on developing a role-based (technical and business) workspace for managing pipelines enhanced by the intelligent framework. Additionally, management-level dashboards and ROI indicators for QA activities were added.

About the author

Albert Tort

Albert Tort

Albert Tort is a software engineering and agile digital assurance & testing passionate with more than 10 years of experience. He is currently the CTO of Sogeti Spain and the country lead of Sogeti Labs. He was a researcher and teacher at UPC-BarcelonaTech where he specialized in requirements engineering, conceptual modelling, quality of information systems and testing. His PhD thesis was titled "Testing and Test-Driven Development of Conceptual Schemas", and he is author of several publications in specialized journals. He has also participated in international conferences. Currently, he also coordinates the postgraduate studies in Software Quality Assurance at the School of Professional & Executive Development of the Technical University of Catalonia.

About Sogeti

Part of the Capgemini Group, Sogeti operates in more than 100 locations globally. Working closely with clients and partners to take full advantage of the opportunities of technology, Sogeti combines agility and speed of implementation to tailor innovative future-focused solutions in Digital Assurance and Testing, Cloud and Cybersecurity, all fueled by AI and automation. With its hands-on ‘value in the making’ approach and passion for technology, Sogeti helps organizations implement their digital journeys at speed.

Visit us at

Capgemini is a global leader in partnering with companies to transform and manage their business by harnessing the power of technology. The Group is guided everyday by its purpose of unleashing human energy through technology for an inclusive and sustainable future. It is a responsible and diverse organization of 325,000 team members in nearly 50 countries. With its strong 55 year heritage and deep industry expertise, Capgemini is trusted by its clients to address the entire breadth of their business needs, from strategy and design to operations, fueled by the fast evolving and innovative world of cloud, data, AI, connectivity, software, digital engineering and platforms. The Group reported in 2021 global revenues of €18 billion.
Get the Future You Want!