State of AI applied to Quality Engineering 2021-22
Section 4.1: Automate & See

Chapter 4 by Eggplant, part of Keysight Technologies

Testing above and below the application surface

Business ●●●○○
Technical ●●○○○

Listen to the audio version

Download the "Section 4.1: Automate & See" as a PDF

Use the site navigation to visit other sections and download further PDF content


By submitting this form, I understand that my data will be processed by Sogeti as described in the Privacy Policy.

Advances in AI have enabled us to close the gap between skilled human testers and UI automation. Now we can test both faster and smarter thanks to a hybrid technique that combines computer vision, intelligent object recognition, and self-generated test cases.

If you could only run one more test, what would it be?

In the previous chapters, we’ve learned about the emergence of UI automation and some of the technologies that have made it possible. Much of the work done to date on UI automation has been an attempt to elevate automation to the level of a skilled human tester. However, to truly bring UI automation beyond mere parity, we must take full advantage of the things that computers are good at.

The case for UI automation is clear. Given the growing complexity of applications and the myriad technologies they utilize and interface with, it’s clear that we can no longer rely solely on traditional code-based test automation. Instead, we should take a hybrid approach that combines the advances in computer vision with more traditional means, such as looking at APIs and object properties. This approach means getting the best of both worlds: the capacity to ‘see’ and interact with complex interfaces combined with an ability to look beneath the surface.

This technique allows for massive productivity gains for testing teams. For instance, a retailer with annual revenue in excess of $4 billion reduced manual testing from 106 hours to just three hours using automation, a 97 percent reduction. This now enables them to run four testing cycles on their omnichannel retail platform during each two-week sprint, compressing 424 hours of manual testing into just 12 hours over the course of the sprint.

UI automation is critical to future-proofing testing capabilities as applications evolve over time. Mimicking real users’ interactions through the UI is vital for answering the question: is this likely to break in the hands of the end user? A hybrid approach can also address another major concern for QA teams: will changes to the application break my tests?

Best of both worlds

Enterprise applications are a perfect example of the complexity of modern software — and the difficulty of testing it. Take Salesforce, for example. With a market share of nearly 20%, it is the biggest CRM vendor. Salesforce is highly customizable. There are hundreds of apps and plugins, custom workflows and rules, user roles and permissions, and a variety of platforms and devices to consider. Because of this, no two companies will have the same deployment. Naturally, this presents a challenge for those responsible for keeping everything running smoothly.

Anyone who’s worked with Salesforce knows that there are two versions: Classic and Lightning. It should come as no surprise that they are coded completely differently. When switching between versions, the entire underlying object library changes. Let’s compare the object properties and code of two identical functions in each version:


Figure: Creating a task in Salesforce Classic


Figure: Creating a task in Salesforce Classic


Figure: Creating a task in Salesforce Lightning


Figure: Creating a task in Salesforce Lightning


Because these objects are identified differently, code-based automation tools like Selenium will require different test scripts for each version. And because Salesforce has loading screens which may hide on-screen elements, there is a clear advantage to an approach that uses computer vision and OCR to visually validate the UI. However, there certainly is something to gain from also looking at the code.

Eggplant, which has been doing UI automation for nearly two decades, takes this approach to Salesforce testing. First, Eggplant analyzes the modules in the Salesforce instance, which can be either standard or customized. By extracting the metadata that describes the various fields and customized rules, Eggplant is able to generate an accurate map of the data flows within Salesforce. In turn, this allows the automation software to generate a library of pre-written scripts, or snippets, which are used to drive the testing model. This entire process takes just a few minutes, several orders of magnitude shorter than more traditional methods. This goes beyond standard UI automation, as the tests themselves are being generated by the AI.

Eggplant enabled a major software company to generate thousands of test cases specific to their Salesforce instance in just three minutes, a process that ordinarily takes dozens of hours.

Figure: Eggplant securely connects to the Salesforce environment (bottom right) and then automatically generates test assets and a model (left) to test the system in a Chrome browser (top right).


Figure: Eggplant securely connects to the Salesforce environment (bottom right) and then automatically generates test assets and a model (left) to test the system in a Chrome browser (top right).


The tests themselves are run by Eggplant’s Fusion Engine, which automates the UI through a hybrid of computer vision and code-based techniques. This allows the QA team to validate the user experience, ensuring that not only does the application pass the required tests, but also that the team is made aware of any anomalous behavior encountered during the automated test runs. This UI-level validation is particularly important with packaged applications like Salesforce, because issues primarily arise around data. For example, if Salesforce is drawing data from an Oracle database, the process of transforming data between different database schemas can ultimately cause issues in the front-end. A traditional functional test may not indicate a problem, but the end user will see a strange or illogical value output. UI validation catches these issues before users encounter them.

As you’ll know, test maintenance can represent a significant proportion of the workload of a QA team. Again, Eggplant’s ability to rapidly build a library of test scripts based on the unique Salesforce environment comes in handy. Every time a new regression test needs to be run, Eggplant repeats the process above, extracting the metadata and comparing it to the previous version to see if anything has changed. Test assets are thus automatically updated each time a change is made to the Salesforce instance. Because QAs may not be made aware of changes to custom rules or extensions, this ability to easily compare Salesforce builds and generate test cases means that we can easily answer the question, “What haven’t we tested yet?”

In the same way that Eggplant can extract the metadata from Salesforce to create a model, it follows the same steps to build ‘sub-models’ from individual pages within Salesforce. This means that smaller components can be tested in isolation without needing to run tests against the entire system. Test cases can then be generated based on the mandatory fields within any module or page. For example, if there are 10 mandatory fields with 2 options each within a module, Eggplant can generate 210 = 1024 test cases. Prioritization of which tests to run can be decided by either the tester or the AI, which I will get to below.

We view this method of automatically modeling a complex piece of software and then generating test cases as the future of test automation. But in order to improve quality across a wider range of applications, we can’t rely only on improvements to UI automation. We must also improve the discipline of test design. Thankfully, the same underlying technologies which have enabled these strides in UI automation — AI and ML — can also help us design more effective tests.

I know they’re here somewhere

Have you ever felt the panic of misplacing your keys?

You start looking in the most obvious place: the bowl next to the door. They aren’t there, so you then have a look in the bag you had yesterday. They’re not in the normal pocket, so you check the others. Hmm, no luck. Where next?

This process continues until eventually, you think ‘Aha! I wore my raincoat last night!’ (This author lives in England.) And there they are, just where you left them. Of course! I’m sure you’ve heard the expression, “it’s always the last place you look.” However, up until that point, you go through a process of looking in increasingly unlikely and obscure places until, at last, you’ve found them.

What’s this got to do with software testing? Well, think of bugs as that elusive set of keys. You may have software that is very good at recognizing bugs, just as you may be very good at spotting the light reflecting off the metal of your keys. However, you’re only able to find bugs in the places that you can think to look for them. Just like in a traditional directed test, you’ll just keep looking for those keys in the same bowl or the same bag. The only thing you’ll learn from this process is that your keys are in neither of those places.

In the world of continuous delivery, we have a vanishingly short window of time to find and address bugs. That’s where intelligence comes in. In the case of the missing keys, you need to rely on your own human intelligence. When looking for bugs, we can use artificial intelligence.

Getting our priorities straight

Central to any problem solving is a consideration of constraints. And constraints create a need to prioritize. Prioritizing test cases is something traditionally left to humans to figure out. But in the same way that AI has helped us make strides in UI automation, it also makes it possible to intelligently prioritize what we should be testing and how we should go about it.

This fundamentally boils down to optimizing test cases. In Section 2, we wrote about the concept of the digital twin and the various algorithms that can help optimize test cases. While the bug hunting and coverage algorithms work in different ways, they support the same goal: broadening coverage of the application surface by reducing the reliance on directed testing.

A fundamental concept in AI is the question of optimization problems. In the language of quality engineering, how can we find the most important anomalies in our application without delaying its release? While it’s easy to find some solution to this problem, it’s very difficult to find the best solution. Therefore, given a limited amount of time and computing power, it’s highly unlikely that a human will be able to specify the best set of tests to run. And that’s precisely where AI can help.

Narrowing the gap, widening the circle

Advances in AI have helped us to narrow the gap between skilled human testers and UI automation. We can now not only test quicker than people, but we can also test smarter, thanks to a hybrid technique that combines computer vision, intelligent object recognition, and auto-generated test cases.

This in turn means that a broader set of people can contribute to efforts to drive quality. AI is directly contributing towards a world where quality is everyone’s business, not just a siloed QA team or embedded tester.

Test automation isn’t just about spotting and reporting anomalies. It’s also about knowing where to look. At the beginning, we asked “If you could only run one more test, what would it be?” Thanks to advances in AI-driven test design and execution, you don’t need to answer that question.

About the author

Jaspar Casey

Jaspar Casey

Jaspar Casey is a Product Marketing Manager at Eggplant, a Keysight Technologies company. He has spent a decade bringing new ideas and digital products to market, covering everything from big data to blockchain. His work involves communicating the unique business value of AI-driven test automation.

About Eggplant

Eggplant is now part of Keysight Technologies.

Keysight Technologies Inc. (NYSE: KEYS) is the world's leading electronic measurement company, transforming today's measurement experience through innovations in wireless, modular, and software solutions. With its Hewlett-Packard and Agilent legacy, Keysight delivers solutions in wireless communications, aerospace and defense, and semiconductor markets with world-class platforms, software, and consistent measurement science. The company's nearly 15,000 employees serve customers in more than 100 countries.

Visit us at