Challenge #3: Accuracy of comparison, as it matters to your users
While most commercial Visual Testing tools look to address the above challenges (i.e. creating and updating the baselines) the biggest challenge, and the deal-breaker, is the algorithm used to give the accuracy of comparison.
For years, visual validation was plagued with false positives introducing too many problems into the testing workflow. Pixel difference, the legacy approach, compares screen images for pixel rendering differences. But rendering issues can cause pixel differences without those differences being true errors. A pixel rendered in 24 bit color from a screen capture can differ from another screen capture by 1 bit. Does this difference constitute an error? Another example, font smoothing, can change between browser releases (see figure below).
Ultimately, over time, engineers concluded that pixel difference technology results in too many false positive bugs requiring manual resolution - put simply, pixel matching just does not scale for modern applications, especially those with dynamic content.
Figure: Pixel based comparison shows false positive due to font smoothing of “Total Balance”
To solve this problem, Applitools invented Visual AI. By replicating how the human eye and brain work, Visual AI only highlights the differences a human would notice.
Figure: Trained on +1 Billion images, Applitools’ Visual AI is 99.9999% accurate and not fooled by browser updates
Visual AI can analyze a screen, a UI component or even an entire web page (i.e. stitching together images captured by scrolling the UI). It then abstracts the UI snapshot into identifiable regions to be analyzed, comparing the size and relative dimensions, colors, and content of objects on a page. Trained on over 1 billion images, Visual AI delivers 99.9999% accuracy and makes automated visual validation possible, enabling teams to get complete coverage for the entire UI with a single snapshot, finding defects that could not be found any other way (see figure below).
Figure: Visual AI finds functional bugs as well as bugs that no other technology can