Testing above and below the application surface

Advances in AI have enabled us to close the gap between skilled human testers and UI automation. Now we can test both faster and smarter thanks to a hybrid technique that combines computer vision, intelligent object recognition, and self-generated test cases.

If you could only run one more test, what would it be?

In the previous chapters, we’ve learned about the emergence of UI automation and some of the technologies that have made it possible. Much of the work done to date on UI automation has been an attempt to elevate automation to the level of a skilled human tester. However, to truly bring UI automation beyond mere parity, we must take full advantage of the things that computers are good at.

The case for UI automation is clear. Given the growing complexity of applications and the myriad technologies they utilize and interface with, it’s clear that we can no longer rely solely on traditional code-based test automation. Instead, we should take a hybrid approach that combines the advances in computer vision with more traditional means, such as looking at APIs and object properties. This approach means getting the best of both worlds: the capacity to ‘see’ and interact with complex interfaces combined with an ability to look beneath the surface.

This technique allows for massive productivity gains for testing teams. For instance, a retailer with annual revenue in excess of $4 billion reduced manual testing from 106 hours to just three hours using automation, a 97 percent reduction. This now enables them to run four testing cycles on their omnichannel retail platform during each two-week sprint, compressing 424 hours of manual testing into just 12 hours over the course of the sprint.

UI automation is critical to future-proofing testing capabilities as applications evolve over time. Mimicking real users’ interactions through the UI is vital for answering the question: is this likely to break in the hands of the end user? A hybrid approach can also address another major concern for QA teams: will changes to the application break my tests?

Enterprise applications are a perfect example of the complexity of modern software — and the difficulty of testing it. Take Salesforce, for example. With a market share of nearly 20%, it is the biggest CRM vendor. Salesforce is highly customizable. There are hundreds of apps and plugins, custom workflows and rules, user roles and permissions, and a variety of platforms and devices to consider. Because of this, no two companies will have the same deployment. Naturally, this presents a challenge for those responsible for keeping everything running smoothly.

Anyone who’s worked with Salesforce knows that there are two versions: Classic and Lightning. It should come as no surprise that they are coded completely differently. When switching between versions, the entire underlying object library changes. Let’s compare the object properties and code of two identical functions in each version:

Figure: Creating a task in Salesforce Classic

Figure: Creating a task in Salesforce Lightning

Because these objects are identified differently, code-based automation tools like Selenium will require different test scripts for each version. And because Salesforce has loading screens which may hide on-screen elements, there is a clear advantage to an approach that uses computer vision and OCR to visually validate the UI. However, there certainly is something to gain from also looking at the code.

Eggplant, which has been doing UI automation for nearly two decades, takes this approach to Salesforce testing. First, Eggplant analyzes the modules in the Salesforce instance, which can be either standard or customized. By extracting the metadata that describes the various fields and customized rules, Eggplant is able to generate an accurate map of the data flows within Salesforce. In turn, this allows the automation software to generate a library of pre-written scripts, or snippets, which are used to drive the testing model. This entire process takes just a few minutes, several orders of magnitude shorter than more traditional methods. This goes beyond standard UI automation, as the tests themselves are being generated by the AI.

Eggplant enabled a major software company to generate thousands of test cases specific to their Salesforce instance in just three minutes, a process that ordinarily takes dozens of hours.

Figure: Eggplant securely connects to the Salesforce environment (bottom right) and then automatically generates test assets and a model (left) to test the system in a Chrome browser (top right).

The tests themselves are run by Eggplant’s Fusion Engine, which automates the UI through a hybrid of computer vision and code-based techniques. This allows the QA team to validate the user experience, ensuring that not only does the application pass the required tests, but also that the team is made aware of any anomalous behavior encountered during the automated test runs. This UI-level validation is particularly important with packaged applications like Salesforce, because issues primarily arise around data. For example, if Salesforce is drawing data from an Oracle database, the process of transforming data between different database schemas can ultimately cause issues in the front-end. A traditional functional test may not indicate a problem, but the end user will see a strange or illogical value output. UI validation catches these issues before users encounter them.

As you’ll know, test maintenance can represent a significant proportion of the workload of a QA team. Again, Eggplant’s ability to rapidly build a library of test scripts based on the unique Salesforce environment comes in handy. Every time a new regression test needs to be run, Eggplant repeats the process above, extracting the metadata and comparing it to the previous version to see if anything has changed. Test assets are thus automatically updated each time a change is made to the Salesforce instance. Because QAs may not be made aware of changes to custom rules or extensions, this ability to easily compare Salesforce builds and generate test cases means that we can easily answer the question, “What haven’t we tested yet?”

In the same way that Eggplant can extract the metadata from Salesforce to create a model, it follows the same steps to build ‘sub-models’ from individual pages within Salesforce. This means that smaller components can be tested in isolation without needing to run tests against the entire system. Test cases can then be generated based on the mandatory fields within any module or page. For example, if there are 10 mandatory fields with 2 options each within a module, Eggplant can generate 210 = 1024 test cases. Prioritization of which tests to run can be decided by either the tester or the AI, which I will get to below.

We view this method of automatically modeling a complex piece of software and then generating test cases as the future of test automation. But in order to improve quality across a wider range of applications, we can’t rely only on improvements to UI automation. We must also improve the discipline of test design. Thankfully, the same underlying technologies which have enabled these strides in UI automation — AI and ML — can also help us design more effective tests.