Log mining for testing with AI

Which test suite is optimal for a new build? That is, in an ideal world? With an infinite supply of time, money, and people? It would involve logically applying user flows from production to the new build, along with any new feature user flows and validations, and observing the result. If we could accomplish this, we could create the ideal regression test for each new build. Observing results through the eyes of real users in a previous release.

The W3C log format (and related IIS, NCSA, Splunk, Sumologic and others) provides insights (most for security) to server activity in production. Log files may span days or weeks and be larger than 50GB.

Figure: W3C format example

The production logs, as shown above, provide solid (and enormous) data showing requests made of the server. This alone is not enough to create much of anything (from a test perspective) since we have no binary information nor any context such as page or page state or even how this request was initiated.

Some background on server requests of note which is that a user action, on average, will make 200 requests of the server to fill the next page or screen. There are exceptions, such as same page applications, where requests may be limited to AJAX or single requests for data. But even in single page applications data is requested from the server upon user actions in most or all cases.

So how can we get context? We could first create an application blueprint (formerly called a Master Key File) that would hold the “keys” to every single possible user action on every possible page state, and their resulting server requests. This would be a very smart AI system of robots which can navigate an application logically, completing fields and marching forward, capturing every action and page and state and server requests.

For example, let’s assume a simple submit button. By submitting a form with specific data, that data will be sent to the server and a variety of requests will be made of the server to deliver certain new elements and results based on that data. Here we see 200 requests made when that submit button is selected by a user:

This example is a great way visually to understand what 200 unique requests looks like compared to that one user action. That set of requests turns out to be quite unique from other actions on this page. And we can use this fact to help us determine what UX flows a user has taken.

By recording every possible UX action in the AI driven blueprint, we group these requests into groups associated with unique specific actions on page states.

Later, we can begin interrogating the log files using an AI computational technique called predator prey. The goal here is that a log file has almost unlimited possibilities of gets puts posts and other requests recorded from users’ production interactions. It’s not atypical for a log file to be 50GB or more, representing 10’s of thousands of user flows and perhaps millions of server requests. As a result, the problem of locating a similar grouping that corresponds to the groups visible in the new build-related blueprint file appears to be infinite. Thus, a technique that is constantly looking for the top three candidates for a match and rapidly discarding the poor matches is extremely useful. Once we have found the best match (there may be no perfect match since the production build is slightly different than the new build), the API group is matched to a UX action in the new build. This is fed to a code generator to generate JavaScript which will execute a UX action which corresponds to the closest match API group for that page state. Typically, we can create 5,000 scripts mimicking production user flows in about 10 minutes. And a QA engineer may decide to set various limits on this such as 1,000 scripts or even 100,000 scripts.

The “AI” here is really the learning which occurred by learning the application through the AI blueprint process, finding similar groups in the logs, and upscaling those API requests to corresponding UX actions in script form. The learning from AI blueprint is also retained for updating the baseline, highlighting differences between builds and so on. Thus, the machines learning is ongoing and forever, but targeted solely at your application.

Figure: Bell curve of usage

The result is most often a bell curve of usage (that is hundreds or thousands of scripts) gleaned from production activity which are then applied to the new build.

In addition to the scripts, we will want to be able to validate other outcomes beyond working flows. This is also a learning process by leveraging these user flows with relevant test data to attain outcomes from these scripts against production. In this case, the AI blueprint can be used from a past run (the production build) to be sure and generate the correct scripts for production or simply use those scripts from that build already generated by the system. Then outcomes can also be recorded and reused against the new build to be certain that outcomes are validated as well as flows working perfectly.

This method has been successfully used across hundreds of applications, validating flows and outcomes as well as all API responses from the server side. Without a single script written by a human.

As the system learns from each build, there is no maintenance of scripts. There is no concern about accessors because the system will automatically apply new accessors to new scripts in each new build. There is no concern that important user flows are skipped because all important flows must be present in production (save new features). And there is no security concern because nothing is added to the application to accomplish this.

Many more details on exactly how all this works can be found in Appvance’s patent 10,204,035.

By learning from both the new build and the previous production release (using AI) and combining this information with standard log files, a QA team can generate the bell curve of user flows that mimic those of real production users and apply those tests to the new build using an AI system such as AppvanceIQ. All of this is accomplished without writing a single line of code. Appvance first made this technology available in 2017.