From Logs to Tests: Usage-driven Regression Testing

Pain point 1: loss of relevance

Automated regression tests tend to lose relevance over time. Initially derived from the analysis of business cases, these tests gradually lose their relevance over time. Because the devil is in the details, sequences of business actions that were worth testing six months ago have lost relevance in light of our customers' actual use of our product. Of course, there is also the issue of relevance loss as a result of the well-known "pesticide paradox" (basic knowledge in software testing - see ISTQB Certified Tester at foundation level). Due to the tests being run on the same routes every time, their ability to detect anomalies goes steadily down. This gradual decline in relevance results in a significant loss of value for the Agile team, with automated regression tests providing ever-poorer feedback. 

Pain point 2: obscure coverage

How comprehensive is our functional regression test suite's coverage? Is there anyone in the team who is aware of the level of coverage provided by our regression test suite? Sadly, the answer to both these questions is frequently No. As one Test Manager from a large insurance company put it: "Automated functional regression tests are something of a black box, and we rarely investigate what they do and cover. It has developed into a significant issue in our team, with doubts about the true value of these tests".

Pain point 3: a disproportionate maintenance effort

This directly results from pain point number two. Due to the team's lack of visibility into test coverage, it's quite difficult to determine where to focus efforts to increase the relevance of automated functional regression tests. Why isn't the team optimizing these tests on a continuous basis to increase their relevance and value? Frequently, the response is (ask your squad): “we don't have the time, and we're unsure where to focus our efforts because we lack visibility into the coverage of critical customer journeys”. The team fixes failed tests (often for technical reasons), but this is typically where the effort ends.

All improvement and problem solving begin with knowledge. Being able to quantify test coverage in relation to our product's most frequent usage is critical for optimizing tests and focusing the team's efforts on the value delivered in operation.

Figure below shows usage and test traces analyzed from the logs in the Gravity tool. Logs in operation (in purple) and logs of test sessions on the test environment (in pink). The overlay on the coverage of user actions is extremely informative to identify missing tests and updating the test suite for the purpose of improving usage coverage.

Figure: Visualization of usage coverage by a regression test suite

Thus, analyzing production and test logs enables us to optimize our regression tests by:

• exploring usage traces and analyzing situations (for example, anomalies in production)
• identifying usage traces to be covered
• generating the necessary automated scripts

Test automation engineers/developers benefit from coverage analysis, while product owners and business analysts benefit from visualizing customer journeys and associated data.

This chapter discusses the various challenges and solutions associated with the creation and maintenance of regression tests leveraging production logs.

Monitoring tools such as Datadog, Splunk, Dynatrace, Application Insights, or Kibana are typically used to manage logs. It is the data source: first, we must ensure that the logs contain sufficient detail regarding user actions (for UI tests) or API or microservice calls (for API tests). Since the artefacts we target are tests (functional regression tests), the logs must capture all actions/calls along with their associated data, allowing for the analysis of complete usage traces.

The following paragraphs detail the 4 main challenges, ranging from data preparation to automated test script generation, within a dedicated framework. Postman is a popular test automation framework for API testing. Numerous frameworks, such as RobotFramework, Cucumber, and others, are currently used for UI testing.

Challenge 1 - From log events to traces

A little terminology is required at this point. A log file is a chronological record of events that occurred during an execution. A "usage trace" is a series of events that represent a functionally consistent usage session.

In this example, Figure to the left depicts an extract of a log file in JSON format. The processing scheme described in Figure to the right is used to differentiate usage traces from logged events. This is the first instance of automated processing in which machine learning assists us in determining the coherent chaining patterns necessary to compute the traces.

Challenge 2 - Explore and analyze customer traces

The second challenge is how to assist the Agile team in exploring and analyzing the traces. Numerous analyses of this data may be of value:

• Clustering the traces generates coherent groupings that aid in trace analysis. On the left, in the figure below illustrates the application of hierarchical clustering to organize the traces.
• The workflow visualization of the traces provides an overview that can be zoomed in to focus on a specific subset of the call graph. The workflow for a selection of usage traces is shown in the figure below, top right.
• Each event in a trace should have its associated data available for browsing and analysis. At the bottom right ofthe figure below, a tree view of the data is shown.

Figure: Visualization of traces and related data

The exploration and analysis of the traces will enable the team, particularly the testers, to dig into an anomalous situation and identify previously unknown user flows (that would be valuable to test in regression because they correspond to customer behavior).

Finally, visualizing the frequency of use of specific user flows enables you to prioritize the execution of tests and also evaluate the impact of an anomaly on these flows.

Challenge 3 - Identify missing/unnecessary tests and optimize coverage

We can calculate test traces from the logs generated during test execution in a testing environment. This enables the determination and visualization of the coverage provided by the current test runs. This coverage is depicted in the figure below, with purple indicating user flows that are covered by one or more tests and pink indicating flows that are not covered.

Testers can then decide whether to complement (or not) existing test coverage, identify unnecessary tests, and optimize coverage through regression test suite updates. Iterative processes are used to achieve the desired level of usage coverage.

The figure below depicts the results of the coverage analysis in two distinct formats (list format and graphical workflow format). This is accomplished by comparing operational usage traces to test traces generated during test execution. Existing tests do not cover the pinked user action sequences.

Figure: Visualization of usage coverage by the tests.

Challenge 4 - Produce and maintain test scripts in your favorite framework

Finally, the final challenge is to generate executable automated test scripts based on the usage traces that must be covered. This is accomplished by automatically generating scripts in the format required by a particular test framework:

• A mapping between the steps of the traces and the REST API calls. This step is facilitated by importing an OpenAPI specification file (in Swagger for example), if it is available.
• Automated generation of scripts, based on keywords or API calls.

The process can finally be summarized in main groups of activities that are illustrated in the figure below.

Figure: The process in 4 activity groups "from logs to tests”

The Gravity tool's support and automation of these activities are based on two types of technologies, namely Machine Learning and Model-Based Testing, which together comprise Smartesting's core competencies. Let's take a closer look at these technologies, paying particular attention to the quality of input data required by machine learning and automated test generation algorithms.

1. Data acquisition & usage analysis - The input data consists of production logs. The primary issue that occasionally arises is a possible lack of precision in the calls (regarding data values, for example). By and large, this data is of high quality. The data comes from commercially available application monitoring tools such as Datadog, Splunk, Dynatrace, or Kibana. Integration with such products for the purpose of extracting logs is relatively simple. Clustering algorithms such as KMeans and hierarchical clustering are primarily used, as they are unsupervised algorithms capable of detecting usage patterns and traces within large sequences of log events.
   
   
2. Visualize & complete test coverage - The algorithms for visualizing traces as graphical workflows (sequences of actions, API calls, and microservices) are derived from Model-Based Testing and are based on abstraction calculations and action/call factorization. These techniques are scalable, meaning they can process thousands of traces simultaneously. Tests can be completed by selecting the traces to be covered. Additionally, we are experimenting with Deep Learning algorithms derived from NLP (Natural Language Processing), specifically attention network algorithms (such as "transformers").
   
   
3. Test generation (scripts, keywords, data) - Additionally, the technologies used in this case are based on Model-Based Testing. It performs filtering and mapping operations on existing scripts, keywords (or Postman collections for API testing), and trace steps. This enables the production of test automation artifacts to be automated to a degree of greater than 80%. In the case of API tests, Swagger specifications provide valuable input for further automating the process (if these specs are detailed enough).
   
   
4. Test execution prioritization - Prioritization is performed using machine learning algorithms that combine reinforcement learning and decision tree learning. The learning phase and prioritization calculation are informed by (1) the history of execution results, (2) test characteristics (complexity, date of last update, execution time, etc. ), and (3) characteristics of the changes made to the software during a build (modified package, granularity of change).

Testing from usage has two benefits for Agile and DevOps teams: it ensures that regression tests adequately cover usage and simplifies/accelerates test maintenance.

Additionally, the various activities of usage trace analysis and coverage completion help to strengthen collaboration between the team's various roles, including the operations engineers. This expedites the analysis of production-related anomalies and optimizes the correction cycle.

The use of machine learning algorithms in conjunction with techniques for automatic test generation from Model-Based Testing enables the production and maintenance of regression tests to become increasingly automated.

This paves the way for the development of an intelligent and autonomous regression testing robot that will simplify life for digital teams in the future.