Image: Playground Games

Traditional game testers might be out of a job in the coming future thanks in part to Xbox’s Matt Booty, who recently revealed that he has been asking researchers at Microsoft to come up with an AI bot for testing games, one that could eliminate the need for human QA testers.

  • “What I always say when I bump into the AI folks, is: ‘Help me figure out how to use an AI bot to go test a game,'” Booty mentioned during a Q&A panel at PAX West 2022.
  • Using AI for QA purposes would be “transformational,” according to the executive, who gave an example of being able to start up “10,000 instances of a game in the cloud” for an AI bot to test all night.
  • Booty currently serves as the head of Xbox Game Studios, overseeing games from Rare, Turn 10, Mojang, and more.
  • Activision Blizzard, which was prompted to make its US-based QA testers full-time employees this summer and give them raises following a unionization campaign, is one publisher that may show interest in the alternative testing method.

From a VGC report:

“Some of the processes we have, have not really kept up with how quickly we can make content,” Booty said. “One of those is testing.

“You think about a game, one of the biggest differences between a game and something like a movie, is if we’re working on a movie and you come in and say ‘hey, this ending, let’s tighten this up, let’s edit this, let’s cut that scene’, it usually doesn’t break anything at the beginning of the movie.

“But in a game you can be ready to ship, and a designer’s like, ‘I’ve got this one little feature, I’m just going to change the colour on this one thing’’ and then it somehow blows up something and now the first 10 minutes of the game doesn’t play.

“So that testing aspect, every single time anything new goes into a big game the whole game has to be tested, front-to-back, side-to-side.

“My dream – there’s a lot going on with AI and machine learning right now, and people using AI to generate all these images.

“What I always say when I bump into the AI folks, is: ‘Help me figure out how to use an AI bot to go test a game.’

“Because I would love to be able to start up 10,000 instances of a game in the cloud, so there’s 10,000 copies of the game running, deploy an AI bot to spend all night testing that game, then in the morning we get a report. Because that would be transformational.

“I always kind of laugh a little bit, people always say ‘the game shipped on Tuesday but I hear they were fixing bugs on Saturday night’ – there’s months of testing and things that have to happen before a game goes out.”

Source: PAX West 2022

Go to thread

Don’t Miss Out on More FPS Review Content!

Our weekly newsletter includes a recap of our reviews and a run down of the most popular tech news that we published.

5 comments

  1. That's not how AI works. Teaching an AI (or what is commonly called an AI today) to play a specific game possibly takes more effort and time than having humans test it. Besides any change to the game will make the AI data obsolete necessitating the purge of the learned data. The worst part is that in order to start teaching the AI how to play a game the game needs to be fully playable first. While a human can test subsystems independently even if some features are not even implemented as placeholders.

    And as stated above, games often break when players do unintended things in them. If an ai is reinforced to play the game as intended it will never detect bugs that occur when you are not playing the game to complete the task intended by the designers.
  2. I think qa testers will be substituted by ai . its a repetitive task that benefits from multiple instances in parallel. I don't see why not an ai can't try all manner of scenarios, and would do so relentlessly and can do so simultaneously many times over. Game and driver testing will surely be eliminated as human jobs, no question. AI brute force I think is perfect for this job.
  3. I think qa testers will be substituted by ai . its a repetitive task that benefits from multiple instances in parallel. I don't see why not an ai can't try all manner of scenarios, and would do so relentlessly and can do so simultaneously many times over. Game and driver testing will surely be eliminated as human jobs, no question. AI brute force I think is perfect for this job.
    Because there is no clear parameters for testing. The AI would need to learn what is intended and what is unintended (aka bug), but in order to do that, it needs loads of samples. Bugs are freak occurences, there is no way to effectively teach an AI to recognise something like that.

    Short of the kind of bugs that would crash the game I don't think current machine learning type AI is capable of replacing human QA testers.
  4. I could see it doing some of the stupid testing that is more or less automated already -- character getting stuck in maps, errant clicks causing unintended results, model collision issues, etc.

    But yeah, I can't really see it totally replacing human testing.

Leave a comment

Please log in to your forum account to comment