World Quality Report 2022-2023: The quest for value through automation

We’ve been researching and writing about automation in the quality assurance landscape for many years. Over that time, both the automation tools and the skills needed to use them have continued to evolve. In this year’s World Quality Report (WQR), we look at why automation can no longer be seen as a separate component of quality but should be baked into the end-to-end testing and development cycle.

What factors determine test automation decisions? We were surprised to find that the return on investment in automation we expected to be a leading determinator was, in fact, not a high priority. Only 26% of survey respondents ranked it in their top three, compared with maintainability (46%), business needs (44%) and new technology (43%), which ranked the highest. This suggests that conversations are no longer about how much an automation tool costs, but what value it will bring to the business.

Lack of benefits delivered

While test automation is now firmly part of the quality assurance and testing regime, our WQR survey revealed a disappointing percentage of team members achieving benefits from it. It seems that expectations are being met only in just over half of the areas assessed, leaving big benefit gains still to be realized. For example, a reduction in test team size due to automation was seen by only 53.4% of team members, while an increase in test coverage was achieved by just 53.3%. Even a faster release cycle courtesy of automation was cited by only 51.7%. At 55.1%, continuous integration and delivery scored the highest, but this was still below expectations.

So, what’s needed to ensure automation delivers expected benefits? This chapter of the WQR recommends getting the processes right, and having clear expectations, good requirements, and a team that can get behind them. We can see that some teams are getting it right, with the positives of automation visible across the testing cycle in key stages. Respondents said they receive most benefit in development (48% placed this in the top three), requirements (47%) and functional testing (46%). At the bottom of the scale, in live (23%) and pre-go-live (20%) are deemed to receive least benefit from automation.

Why automation is needed

We believe that automation is critical to delivering at the speed and quality we all need to succeed in the modern world. To this end, the WQR offers a set of recommendations for increasing the value from quality automation initiatives. These include starting automation as the requirements are being created by building in an automation-first approach, and getting automation requirements agreed before you start to automate. Tooling and frameworks should be reviewed on a regular basis and remember that one tool doesn’t do everything, so pick the best one for the job.

Get in touch

If you’d like to hear more about our findings relating to test automation and why it needs a three-year roadmap, please get in touch with: