State of AI applied to Quality Engineering 2021-22

Section 2: Design

by Sogeti

Business ●●●●●
Technical ○○○○○

 

Listen to the audio version

Download the "Section 2: Design" as a PDF

Use the site navigation to visit other sections and download further PDF content

By submitting this form, I understand that my data will be processed by Sogeti as described in the Privacy Policy.*

Introduction

One of the most critical objectives of quality engineering is to provide clear advice on quality and risk in a manner that instills confidence in all parties involved. To accomplish this, we must collect information about the system's behavior. This includes, but is not limited to, system requirements, functional design, user manuals, and/or administrative procedures. This activity, known to us as test design, is the process of developing, writing, and updating test cases in response to new or changing requirements. The goal of test design is to make testing more systematic and responsive to change. It should enable rigorous testing that fits within the same cycle, same sprint, while still allowing stakeholders—from business analysts to developers and testers—to remain aligned and flexible.


There are numerous test design techniques. To assist in determining which techniques to employ in a given situation, TMAP defines four so-called coverage groups [1]. Each of these groups can be assigned to a particular test design technique. Additionally, TMAP strongly encourages to always combine coverage- and experience-based testing (such as exploratory testing).


Developing the appropriate set of test cases is critical for connecting the test strategy and its implementation - the tests that are executed. A test case is used to determine whether the system behaves as expected under specified conditions. It must therefore contain all of the components necessary to generate that system's behavior and determine whether it is correct.  A well-known way to describe system behavior is
‘Input → Processing → Output’.

The following diagram illustrates the relationship between the terms used in test design:

Implementing the test strategy

Source: https://www.tmap.net/media/91

[1] A coverage group is a group of coverage types and test design techniques that aim at testing the same aspect of an IT system or business process. The four coverage groups are: process, condition, data and appearance.

Test design challenges

While test design is critical for addressing basic questions such as "which test cases?", "how many test cases?" this activity has become more difficult than ever. Poorly defined requirements or user stories, and complexities of maintaining multiple versions of requirements lead to coding errors and in-sprint testing inefficiencies.

  • Around two-thirds of all respondents of the 2020 Continuous Testing Report said that designing and maintaining meaningful test cases that align with user expectations is a significant challenge.
  • Some respondents have been sharing difficulties associated with understanding and authoring test cases from text-based requirements, with the risk of accidentally missing or designing incorrect test cases.
  • Over half of respondents stated that they typically end up with far more test cases than necessary, with significant overlap. They appear to be modeling tests rather than requirements.

Organizations conduct excessive testing in some areas and insufficient testing in others. Appropriate test design techniques should provide the necessary coverage and allow to write appropriate test cases, and make test automation easier. Assistance in test design is very much required.

This section discusses how artificial intelligence can be used to address some of the challenges associated with test design.

In this section

Chapter 1: Moving to an AI-driven digital twin

Chapter 2: AI as co-creator of test design

Chapter 3: The power of process intelligence

Chapter 4: Dynamic optimization of test cases

Chapter 5: Another view on test case optimization