Chapter 1 by Sogeti
Business ●●●○○
Technical ●●○○○
Listen to the audio version
The delivery of IT solutions with a quality focus requires intelligence, which can be augmented by the use of smart quality gates. Such gates can be gradually deployed in accordance with the maturity DevOps journey.
We all know that digital business requires both efficiency and a quality-driven mindset. While Jez Humble's CALMS (Culture, Automation, Lean, Measurement and Sharing) framework[1] can help us pave the way to more efficient delivery, the pursuit of quality necessitates a continuous testing and quality assurance strategy[2]. Both of these principles must be pursued within a context of limited resources, which makes it critical to constantly analyze, reason, prioritize, and choose between alternatives.
As writer John Ruskin put it, “Quality is never an accident. It is always the result of intelligent effort”. Indeed, the term "intelligence" derives directly from the Latin words intelligentia and intelligere, which mean “to understand, comprehend, come to know” and it is the combination of two primary words: inter (between) and legere (choose). In summary, delivering IT solutions takes intelligence to maintain a constant balance of risk and quality when delivering business value[3].
In this chapter, we propose a strategy for developing an intelligence base to support and enrich project quality gates. Naturally, a set of enhanced metrics is also needed to augment our intelligence, which we will examine further in the next chapter.
[1] R. Marselis, B. van Veendendaal, D. Geurts y W. Ruigrok, Quality for DevOps teams, Sogeti, 2020
[2] Sogeti, «Continuous Testing Report,» 2020
[3] A. Tort, «Is (Artificial) Intelligence Needed for Testing?,» SogetiLabs blog, 2018
The creation and evolution of intelligent workspaces is critical for efficient and quality-driven IT delivery, and it is now possible due to increased computational capabilities (based on the well-known "Moore Law") that allow the application of narrow artificial intelligence to the quality assurance domain for software projects[4]. The 2020 World Quality Report[5] concludes that use cases such as “use predictive test selection and optimizations”, “optimize test schedule and effort in a risk-free manner”, “release risk prediction” or “automated defect remediation”. To improve our decision-making process, we must acquire and apply knowledge. Acquiring knowledge in IT projects entails gathering data, converting it to information, and using it as knowledge to manage the project, as well as a continuous learning process that accumulates knowledge as experience.
Such intelligence can be fragmented and implicitly stored in the brains of human project professionals. Owing to human memory constraints, excessive knowledge contexts, ambiguity, or the need to act fast[6], this often results in intuition, subjective disputes, and cognitive bias that violate the "what you see is what there is" (WYSIATI) requirement. By comparison, we recommend making the intelligence process used in DevOps more explicit as a foundation for augmenting human intelligence with artificial intelligence.
DevOps is typically applied by (as much as possible) automated pipelines that define the explicit steps required to move software from development to production. However, as new software contributions evolve to production through development and deployment activities, the risk of defects increases, as each operation can result in a variety of potential defects. As a result, a special type of DevOps pipeline activity called Quality Gates must be defined, implemented, and orchestrated throughout the pipeline.
[4] Sogeti, «AI First: Learning from the machine,» 2017
[5] Sogeti, Capgemini and Microfocus, «World Quality Report,» 2020
[6] D. Kahneman, Thinking, Fast and Slow, 2011
Quality gates are defined as a “certification that all appropriate work required to move products forward -a gate- to subsequent project activities has been completed and reviewed, and product meets specific quality expectations”. Our definition of Smart Quality Gates for DevOps contexts is based on [7]: “Significant milestones and decision points with predefined and quality focused criteria”. However, in many contexts, quality gates have been simply conceived as checklists or formal signoffs that indicate completion of a particular software development artifact. Additionally, these type of quality gates were manually informed and controlled, often using binary passing criteria.
Our approach is diametrically opposed to the conventional approach of a late massive validation step prior to product delivery, and is more akin to a continuous series of checklists (as when operating an airplane) to minimize risks and ensure quality consistency throughout the lifecycle.
By adapting this concept to our context, we describe a Smart Quality Gate as a critical decision point with quality-focused parameters which can be tested automatically using project (human and artificial) intelligence. Such a gate aims at mitigating quality risks as software contributions progress across DevOps pipelines.
Taking this definition into account, the table below states the main components of a quality gate
| Component | Description |
|---|---|
| Entry criteria | The subset of project input data that is required to be considered by the quality gate. |
| Exit criteria metric | A (automatically or semiautomatically) computable measure about the current quality of the project. This computable measure may be a simple input, a simple calculation, or an advanced analytics & AI based model execution. |
| Exit criteria threshold | A configurable threshold (decision value), the meaning of which is directly related to risk reduction. The quality gate will be passed if the exit criteria metric value in each moment is higher than this threshold. |
| Exit criteria measure unit | The unit measure in which the threshold is computed. |
| Severity | It defines the type of reaction when a quality gate fails. The default reaction is stopping the pipeline and refuse the software contribution. An alternative is only providing a warning, but not stopping the pipeline. |
| Notification policy | It defines the notification policy for quality gates passes and fails. |
[7] T. Flohr, «Defining Suitable Criteria for Quality Gates,» de International Conferences on Software Process and Product Measurement, 2008
Quality Gates must be implemented in the corresponding orchestration job server as pipeline steps. We have deployed quality gates into a variety of platforms, like Azure DevOps, Jenkins, Bamboo, and GitLab. Each quality gate must establish connections to the intelligence system in order to meet its admission requirements.
Additionally, the metric's computation must be applied, and its output must be exposed to the Quality Gates configuration and management layer. The pipeline will either proceed or be shut down, depending on the severity. Similarly, the corresponding notification behavior will be activated in accordance with the notification policy.
The core of Smart Quality Gates approach is the setting up, maintenance and evolution of an intelligence system capable of providing information as a service to Quality Gates implementations for use in the DevOps workspace.
The intelligence framework must collect data explicitly and automatically through connectors from the project's various data repositories (test cases, defects, code, user support, etc.) and structure it as a knowledge base model.
Additionally, advanced analytics and cognitive models can be implemented to provide advanced intelligence features that are exposed to the Quality Gates implemented in DevOps pipelines via Application Programming Interfaces (API). Such Quality Gates can also report back exit criteria that can be used to enrich learning procedures within the intelligence system.
The need for a quality gates manager
What we have observed in many DevOps contexts is that quality gates have two clear sides:
We suggest that the management and operation of both sides be explicitly separated, especially because they are assigned to distinct project roles. This is why we advocate for the development of a dedicated layer to handle and monitor quality gates in a non-technical manner, including the configuration of exit requirements thresholds, severity, and notification policy.
The Figure below shows an example implementation made by Sogeti to manage quality gates from a business risks perspective.
Figure. Example screenshot of Quality Gates Configuration & Management Layer
The quality gates manager may be complemented by the implementation of automatic dashboards that provide the big picture about the quality status and evolution of a project, or the whole set of projects in the organization. The implementation of this layer may be done through Sogeti Dashboards but taking most of the built intelligence system as input source.
A phased deployment based on maturity
We implement the smart quality gates approach in an incremental way, based on three dimensions: (1) a maturity journey of the smart quality gates system, and (2) the progressive addition and enriching of smart gates, and (3) the evolution of the intelligent system.
By this way, DevOps environments may evolve through the following stages regarding smart quality gates system:
Smart, integrated and managed QA gates system within DevOps
The figure below shows the general overview of our Smart Quality Gates approach. The main aim is to set-up a working environment to deliver IT value based on:
This approach is also layered-based with the aim of associating both technical and business professionals. These are the main components of the approach:
Figure: Smart Quality Gates approach overview
Each analytics and AI-based model requires an standardized API structure to be integrated into the intelligence system, consisting of a set of methods that may be used in the technical implementation of pipeline steps, including the required input values and the expected outputs. Each API method became a usable intelligence service chunk to be used in pipelines to build the quality gates, but it is expected to include the following items:
Each intelligence service chunks (descriptive, prescriptive and predictive) constitutes the evolving QA brain for automatically supporting projects and enhancing quality gates to be smarter and integrated to the DevOps workspace.
Use Case: Building a progressive smart QA workspace in the water management sector
Smart QA Gates is introduced incrementally over a three-year period for one of the world's leading environmental service companies in the water management market. Three significant obstacles had to be overcome:
To address these issues, a QA maturity plan was developed and accompanied by a phased implementation of smart QA gates:
Albert Tort
Albert Tort is a software engineering and agile digital assurance & testing passionate with more than 10 years of experience. He is currently the CTO of Sogeti Spain and the country lead of Sogeti Labs. He was a researcher and teacher at UPC-BarcelonaTech where he specialized in requirements engineering, conceptual modelling, quality of information systems and testing. His PhD thesis was titled "Testing and Test-Driven Development of Conceptual Schemas", and he is author of several publications in specialized journals. He has also participated in international conferences. Currently, he also coordinates the postgraduate studies in Software Quality Assurance at the School of Professional & Executive Development of the Technical University of Catalonia.
Part of the Capgemini Group, Sogeti operates in more than 100 locations globally. Working closely with clients and partners to take full advantage of the opportunities of technology, Sogeti combines agility and speed of implementation to tailor innovative future-focused solutions in Digital Assurance and Testing, Cloud and Cybersecurity, all fueled by AI and automation. With its hands-on ‘value in the making’ approach and passion for technology, Sogeti helps organizations implement their digital journeys at speed.
Visit us at www.sogeti.com
Capgemini is a global leader in partnering with companies to transform and manage their business by harnessing the power of technology. The Group is guided everyday by its purpose of unleashing human energy through technology for an inclusive and sustainable future. It is a responsible and diverse organization of 325,000 team members in nearly 50 countries. With its strong 55 year heritage and deep industry expertise, Capgemini is trusted by its clients to address the entire breadth of their business needs, from strategy and design to operations, fueled by the fast evolving and innovative world of cloud, data, AI, connectivity, software, digital engineering and platforms. The Group reported in 2021 global revenues of €18 billion.
Get the Future You Want!