Computer vision (CV) is the ability to extract information from images. Our AI engine understands a screen’s composition and breaks it down into the unique objects that it contains.
In terms of architecture, we implemented the AI engine as a separate module. Rather than restricting it to a specific product, any of our testing products can theoretically use the engine. The first product that integrates with the AI engine is Micro Focus UFT One, our flagship test automation tool.
A UFT One GUI test consists of a test script that contains several test steps. A step is a statement that performs some action on the application under test (AUT). A pseudo-code example might be “Click the shopping cart” or “Enter 2 in the ‘How many items’ textbox.” Note that while a typical test script includes other types of steps such as calculations, or application logic, this discussion pertains to steps involving objects on the screen, such as manipulating them, or performing checkpoints or validations on them.
When UFT One’s test execution engine encounters a step with an object, it presents the object to the AI engine, which invokes its CV algorithm to look for the object on the screen. If it finds the object, the AI engine provides metadata about its location back to UFT One’s test execution engine. It then performs the action required on the object: clicking it, entering text, selecting it, etc.
Most importantly, the AI engine knows nothing about the implementation of the object. It treats the object as an image, regardless of the device or platform it comes from.
The CV capability is supported by an artificial neural network (ANN), a layered structure of algorithms that classify objects. We train our ANN with a large number of visual objects, resulting in a model that identifies objects it will likely encounter in applications under test (AUT). Thus, when the AI engine is tasked with locating a specific object, it utilizes the model to identify a match in the AUT.