It’s 2am on launch night and you’re in the war room. Players all over the world are experiencing your game for the first time – tomorrow, the first reviews will be all over the web. But the game is struggling to scale. You turn to the runbook, but it’s clear almost immediately it’s out of date. What do you do?
It’s a game developer’s nightmare, all too often turned-reality. For smaller studios that don’t have access to the infrastructure of larger companies, rigorous testing ahead of launch is a constant challenge. How do you get sufficient hardware to test your game? How do you effectively scale-test your systems? Where do you get the resource to manage and build your deployment process?
How Midwinter Entertainment pulled off the Scavengers Early Access launch with minimal backend issues
Thanks to a tactical collaboration, the Scavengers Early Access launch avoided “a lot of the classic challenges,” according to Midwinter Entertainment’s Vice President of Technology, Eric Matteson.
“[Working] with IMS, we brought up and down multiple regions on the day of launch and split regions in two,” he explains. “We were able to respond to customer demand almost instantaneously – bringing up regions where we had a higher population than we thought we were going to have, so that we could place game servers and lobby servers in the location where players could have the best latency experience.”
Eager to know more, we caught up with Eric and Jared Dixey-Hefty – Principal Engineer for Scavengers Live Operations – to find out how their teams worked together to improve Scavengers’ gameplay and pull off a successful launch.
Parallel testing got Scavengers to market quicker
Parallel testing meant Midwinter Entertainment could deploy builds in clusters without impacting other builds. By maximising the number of iterations in development, they were able to test multiple scenarios simultaneously and fix things faster. Not only that, it was also more cost-efficient because the same machine was used for various tests.
“In what we call the development environment, it’s kind of the Wild West. Developers can actively make changes – they’re all running alongside each other in an isolated fashion, solving the problems they need to worry about, the things they need to iterate on quickly… That day-to-day experience gives them a nice way to validate,” says Jared.
“Games are fuelled by your ability to understand, diagnose and respond to real player issues and real player updates”
- Eric Matteson - Vice President of Technology, Improbable
Daily playtesting allowed them to see through players’ eyes
Because the internal infrastructure available via Improbable Multiplayer Services (IMS) matches the infrastructure deployed for live production, Midwinter Entertainment were able to constantly playtest internally from early on in the development process – and get realistic results. They also used ‘sim players’ to load-test the system from end to end, giving them the inside track on what real players would see.
When they came across sticky issues, cloud-based infrastructure and global support teams in opposing time zones meant engineers could work in their sleep – or, rather, work got done while engineers slept.
"Games are fuelled by your ability to understand, diagnose and respond to real player issues and real player updates,” explains Eric. “The only way to evaluate a complex, large-scale multiplayer game like Scavengers,” he says, “is to touch it, to feel it, to play it – and to play it the way that players are going to play it."
Red team and blue team QA testing kept runbooks up-to-date
When development is moving fast, it’s all too easy to assume things are working smoothly – simply because you haven’t been alerted otherwise. That’s why they took a two-pronged approach to quality assurance, tasking a ‘red team’ to act as saboteur. “The saboteur’s job is basically to go through common failure points, or dustier areas of the platform that we haven’t really tested as recently, and make sure that stuff is still up to scratch,” explains Jared. Meanwhile, the ‘blue team’ is tasked with fixing the mess they (intentionally) make.
Working on Scavengers, the red and blue teams identified out-of-date runbooks, important alerts that didn’t fire (or fired too often, which any engineer will tell you is almost as frustrating) and systems that didn’t scale as expected. All were caught in time to be fixed ahead of launch – thus avoiding potentially expensive mistakes.
“Players will always do something that you never expected. They will find another way to play the game that’s very different than what you thought.”
- Eric Matteson - Vice President of Technology, Improbable
Community playtesting generated invaluable gameplay feedback
Live closed beta tests with the community meant tens-of-thousands of people played Scavengers prior to launch, rather than a hundred or so in-house. This allowed the team to stress-test more factors, as well as identifying areas for gameplay improvement.
“Players will always do something that you never expected,” says Eric. “They will find another way to play the game that’s very different than what you thought. And without that interaction with customers, with early playtesters, experiments with gamers, you don’t have that opportunity to get that information that allows you to craft a great game.”
Extra support enabled fast updates after Early Access launch
“The world in which you release a game and that game is the finished product – that’s gone,” says Eric. “Now, really, your game development process starts when you start active interaction and constant interaction with players.”
Midwinter Entertainment were able to update Scavengers in four days after Early Access launch – including fixing crashing issues, making quality of life improvements, and implementing a significant balance change. “Games are fuelled by your ability to understand, diagnose and respond to real player issues and real player updates,” says Eric. “Being able to do that in four days is a tremendous advantage.”
Turning your Achilles’ heel into your superpower
Sometimes it’s lack of experience in running online multiplayer games at scale that’s a game studio’s Achilles’ heel at launch. But often it’s just that they’re “focusing all their effort and energy into the game itself,” explains Jared. A studio’s prerogative, surely?
Far from being a vulnerability, obsession with gameplay should be a superpower in such a competitive space. Scavengers is proof that, with a bit of tactical collaboration, it can be.
