Now that you have an understanding of AI for game testing technologies, it’s important to discuss their implementation in practice. Here, I present two anonymous case studies, and share real-world experiences from deploying these technologies to production.
Game Store Testing Case Study
This first case study involves a gaming industry leader that operates some of the world’s largest interactive games and has over 50-plus million customer accounts. This industry leader licenses its 3D graphics technology and visual simulation software to other industries. This 3D technology supports its competitive advantage in gaming, film, and other interactive entertainment venues. The company employs hundreds of software developers worldwide and has offices and facilities across the globe. It subcontracts manual testing overseas and has several hundred testers that regularly provide testing services.
The developers and testers at this company experience constant “ping pong” errors. One engineer makes a small tweak and tests the gaming modules, and then another one finds a new bug. So, perhaps a month later, this developer fixes that bug, and now the first developer notices, as if by magic, a new or similar bug to the one they just fixed a month ago. The testing strategy being used does not adequately address real regression testing of the game end-to-end, and the company views such a complete, end-to-end full regression for games as virtually impossible.
Initially, the company does not quite believe that AI-based tools can provide the claimed automation. However, the testing and quality assurance team of its flagship game feels that legacy tools are such an inhibitor to their speed and efficiency that they decide to invest.
So where is their starting point? Well, since their games are almost entirely graphical, with hundreds to thousands of purchasable items, they began with transforming the testing of their game store—an important revenue source. Changes to items in their game stores happen daily and due to the large scale, these changes are often not adequately tested. The result often leads to customer requests for support and refunds. When there is a product defect, it can create a high volume of customer support calls and frustration on the part of the gamers.
Integrating AI for game testing into this company’s workflow has been transformative. They now have a fully self-service solution where they can define test cases for hundreds of their assets, and train new assets on the fly. The gap between the capabilities of their legacy test automation tools and modern DevOps practices for their game store has been eliminated, and they can now support the high demand for new content quickly and reliably. They went from a 2 week long regression testing period, down to being able to run the same tests within 12 hours. Better yet, if they wish to run those same tests in 6 hours, all they have to do is double the computational resources and so they are able to scale on-demand with technology rather than people. Needless to say this gaming industry leader is now a believer in the capabilities of AI for game testing and has experienced the leap in productivity these technologies can provide first-hand.
Game Stream Testing Case Study
Our second and final case study is the gaming division of one of the largest enterprise software companies in the world. Its products include gaming hardware such as consoles and peripherals, and gaming software, including the hosting of cloud-based digital content that stream games to its customers. These streamed games are developed by both the company’s own software development teams and other third-party vendor studios.
The game streaming service has hundreds of games that need comprehensive quality assurance and testing. A substantial percentage of its applications are updated frequently and require continuous validation to maintain user experience quality. All of this adds complexity to the quality assurance and test process. Performance is essential to the streaming service, and the development team is tasked with finding answers to these tough questions: What is the measured response to an individual player for a click on any particular element? Is this adequate to maintain the satisfaction of the user experience? How can this validation and quality determination be done continuously?
As a brand new release of their game streaming service was approaching, this company knew they needed to find the answers to their questions, and find them fast, and so they also made the choice to invest in AI for gaming technologies. Their main focus was performing top of the rack switch testing for their game streaming data centers. Using AI, not only were they able to identify performance issues when interacting with specific game titles under load, but they were also able to reproduce a network switch issue that was manifesting itself in production environments. The game streaming platform team was successfully able to launch the new release, and reduce 20 person-days of test scripting down to 10 hours of AI-driven test execution. Reflecting on the entire experience, the company reported that they believed that they would not have been able to release in time without this technology.