In building out RACHAEL, we wanted the solution to be able to deliver automatically the crucial definitions, or elements, needed for representative performance modelling:
- The load testing scenarios that would need to be designed
- Different populations and their typical user path
- The think time that would need to be applied
- The pacing
- The load policy — the typical curve it follows
In any AI project, the most important element is the quality of your data. The data needs to have enough details to apply the right algorithm to parse, group the traffic, and generate the user flows identified in the production traffic.
Nowadays there is a ton of data available that can help create a realistic model of real-world scenarios, but it’s spread across multiple data sources (logs, APM, marketing tracking systems, etc.). Sometimes engineers have access to this data, sometimes not. Different scenarios call for different data elements. Either way, it can take weeks to retrieve and cobble together the data that “fits” the scenario you are testing.
Protocol-based load testing requires the exact http traffic to generate a script with the following level of details:
- URL parameters
- Body of the post requests
- http headers
- Session id
For various reasons (storage, data privacy, security), most data sources collect only some of the data necessary to create a representative performance model. That’s the biggest challenge: to get the right data source that has enough details. How to extract user traffic data from production for a specific period of time to generate use cases for the test? How best to capture and leverage user experience data to define precise settings — load policy, think time, pacing — for the test? How to identify realistic locations and actual network conditions?
In a nutshell: how could we use AI to turn actual network traffic into a data source?
NeoLoad evaluated several different AI approaches to see what best fit our goals. All AI solutions are really a combination of different techniques, technologies, and methodologies — there is no universal magical system. An approach that met our overarching-principles criteria was traffic mirroring.
Traffic mirroring is a technique whereby live production traffic is copied and sent to two places: the original production servers and a testing environment. Cloud providers like AWS and Azure already offer this option, and we validated this technology with large-scale users. Essentially, it allows you to work with real-life traffic without real-life impact.
With RACHAEL, we use a small, non-intrusive mechanism that duplicates (mirrors) the traffic that is going in and out of the application in production. The traffic is then stored and analyzed for a representative period of time (say, 8 hours for a workday) to extract the most valuable elements needed for your scenario. You get data on a wide range of usages — how different populations are actually using applications. For each such population, you can determine a representative synthetic user path that, say, 80% of users follow. You can also establish a realistic and accurate load policy (load curve) for each population. Do they spike very quickly from the beginning of the time frame? Or do they go up to a point and then plateau? Mirroring actual production traffic also reveals other important properties such as the think time between each action or the pacing between each session.
Where AI can save a lot of time and headache is defining variables and dynamic parameters. While modern performance engineering solutions such as NeoLoad make it significantly easier to manage parameters with dynamic values through advanced correlation and frameworks, determining which parameters are dynamic and generating data for these variables remain some of the most time-consuming tasks. Not only does AI accelerate the analysis, but mirroring production traffic captures data from the system under test that may be closed to your production environment. With RACHAEL, a dataset is provided for each variable, so the effort of variabilization is drastically reduced. But you don’t have to use these datasets. If test data and production data are not close (e.g., healthcare, insurance), you can simply forgo the AI-generated datasets.
In sum, leveraging production traffic as a data source allows performance testers to extract with precision the crucial elements needed to build realistic, representative scenarios:
- Variables and dynamic parameters, with auto-generated datasets
- Representative user populations, their typical user paths in the application
- Real-life properties such as the think time between each action or the pacing between each session
- How to ramp up these populations to reflect the real production load curve