One such implementation of this technology is Tricentis Vision AI. With this approach, UI elements are identified based on their appearance rather than their technical properties. It makes no difference whether a single UI element is redesigned, or the entire application is rewritten in a new technology. Like a human, the automation will simply adapt and figure it out. By utilizing machine learning to perceive and steer a UI in the same way that a human user would, you can ensure that your automation is as adaptable as the human brain. If you can see it, Vision AI has the capability to automate it. This might be an app that utilizes now-deprecated technologies, an app that utilizes new technologies, or an app that you access remotely. You can even begin automating tests with mockups or whiteboard designs.
This is accomplished through the use of intelligent object detection technology to distinguish user interface elements. While this is a novel technique to software testing, firms like Tesla utilize it to detect objects (other cars, pedestrians, signs, stoplights, and trees) for self-driving cars. Why would you take self-driving vehicle technology and apply it to test automation? Because this addresses the above-mentioned speed and accuracy issue. A self-driving automobile must perceive objects properly in real time. Any delay will result in accidents – and may even result in death. Vision AI processes 40 frames per second using the same quick and precise technique to intelligent object detection as self-driving automobiles (vs 1.8 fps with other tools and 24 fps with the human eye).
In developing Vision AI, we took intelligent object detection technology and adapted it to detect controls and understand user interfaces. Rather than checking for pedestrians, signs, and stoplights, it can look for dropdowns, tables, lists, and menus – in fact, any control that a person can identify. Detecting the controls, however, is simply one aspect of the issue. Additionally, we must read the screen in real time. This is where a novel approach to optical character recognition (OCR) comes into play. OCR has been around for over two decades, yet it is still somewhat slow. Even with industry-leading OCR, reading a screen takes seconds, but navigating UIs like a person requires real-time character recognition that reacts in milliseconds. That is why we created an entirely new category of optical character recognition powered by AI.
However, like with self-driving automobiles, simply seeing and understanding things is insufficient. You also need to drive. Vision AI was never intended to be a “test automation tool,” but rather a “test automation engine.” As such, it must be integrated with something capable of performing basic functions such as test data management and test case design. Consider Vision AI as a way to boost the performance of your existing tool suite rather than requiring an entirely new set of tools. With this additional layer of intelligence, your automation is intelligent enough to operate through the great majority of UI changes – things that invariably trip up traditional automation but would not occur to a human.