Why are Game Developers Never Prepared for Big Launches?
After ‘Halo: Master Chief Collection’ became the latest game to disappoint fans by launching with a broken multiplayer experience, we decided to reach out to some industry experts in an attempt to find out why game developers never seem to be prepared for big launches.
If it was up to us, we would’ve had a number of Halo: MCC articles posted here last week – ‘Map Overviews’, ‘Weapon Guides’, ‘Top 10 Game-Winning Kills’. However, in order to post those types of articles, you need to have been able to play the game… and there lies the problem.
For those not in the know, let us quickly catch you up. Here’s last week in a nutshell:
It’s launch day – Halo: MCC is finally out. We’ve ingested a bountiful amount of caffeine and we’re ready to begin a deep vein thrombosis-inducing gaming session.
We load up the game, only to be greeted by a mandatory 20GB ‘patch’; without it, you can’t play online.
Note: For those not in the know, a patch is generally a small update (usually less than 1GB) that fixes a bug found in a game after it’s been released. You see, in previous years, if a game launched with a bug, there was no real way of fixing it once it was in the hands of the consumers. However, now that a constant connection to the internet is an intrinsic element to console gaming, games can be fixed, tweaked and upgraded years after their release – a minor cyber-miracle.
So, 20GB… seems pretty hefty for just a patch doesn’t it? To put that into perspective, that’s larger than the original file sizes of Halo: CE, Halo 2 and Halo 3 COMBINED. Even with a fibre optic connection, a file of that size would take upwards of 10 hours to download and install.
We’re annoyed, but fine… we can find something else to do whilst it downloads. After all, ten hours is a small investment to make in order to relive those magical multiplayer moments that had us hooked for years.
…ten hours later. We’re knackered, but at least the patch is installed, right? Let’s try to get just one quick game in before bed.
Enter multiplayer mode. Search for game. Sit in lobby for 15 minutes… ‘No games found.’
“People are probably still downloading that patch, I’ll try again tomorrow.”
We accept defeat for the day and go to sleep.
We can now stop describing this affair in such small increments of time. Why? Because the rest of week was entirely spent doing the same thing – searching for multiplayer matches.
The result? We managed to connect to a grand total of 7 multiplayer matches in the whole week… yes, 7. And to top it all off, there wasn’t a single one that wasn’t laden with lag. Simply put, it was unplayable.
This would barely be justifiable for a low-budget, debut title from an indie developer; but for a AAA franchise like Halo, the current state of its multiplayer is just perplexing. Its not even that we’re angry per se – more than anything, we’re confused. What’s the cause? We simply can’t understand how a game of this magnitude, from a developing company who pride themselves on their history of burgeoning the online gaming experience, could completely butcher a linchpin feature of their franchise. It just doesn’t make sense.
And Halo: MCC is not the first game that this has happened to. Last year, the multiplayer mode for GTA: V was almost unplayable for a month. The year before that, Diablo 3 caused an uproar with it’s infamous ‘error 37’. These are big games from big companies – companies renowned for taking pride in their products. So why does this keep happening?
We decided to reach out to videogame industry consultant Ben Cousins to help us wrap our heads around this ordeal. He suggested that Halo: MCC developers, 343 Industries, “didn’t test [the] code properly at scale”. Essentially, he believed that the developers didn’t adequately prepare for the sheer quantity of players that would be simultaneously using their servers – preparing for this is known as ‘stress testing’.
This could be the reason as to why the multiplayer hasn’t been working, but even still, we have to ask ‘why?’.
Surely 343 Industries knew roughly how many people were going to buy Halo: MCC at launch? Couldn’t they have simply looked at the pre-order figures and then said to themselves “So, we have 1 million copies of the game pre-ordered and our sales team expects us to sell another million copies in the opening week – lets get the servers ready for a potential of around 2 million concurrent users*”. Is that not logical? Or are we oversimplifying it? We genuinely want to know. What else could be the cause?
*Of course, it’s unlikely that every single person who owns the game would be trying to play it at the same time, but for the sake of this argument, lets assume that they were.
Okay then, let’s assume that 343 Industries exceeded their projected sales – they still couldn’t have sold more copies of the game than the total amount that they’d manufactured. Let us explain: If they had produced 4 million copies Halo: MCC on disc, there couldn’t have been more than 4 million concurrent users online in the first week. Thus, surely they could’ve prepared their servers to accommodate for that amount of players (even if it was the ‘worst case scenario’). Were they simply not prepared to host a player-base equivalent to the amount of discs that they manufactured? Surely not.
Okay, even if you argue: “343 Industries knew how many physical copies of the game were made, but what about the infinite amount of copies that could have been bought digitally?.” I agree, there’s no real way to predict digital sales like you can with physical ones, but aren’t digital sales are just as easy to quantify? As soon as you purchase a game digitally, the download comes directly from the developers server. Thus, 343 Industries should have known exactly how many digital copies were ‘distributed.’ Why couldn’t they just add the number of digital purchases to the number of physical copies manufactured and then prepared to host that total amount of players? For instance, 343 Industries could have said to themselves – “We could potentially have 4 million players (with a disc) online, plus 1 million players online who have downloaded the game digitally. Let’s prepare our servers for the maximum potential of 5 million concurrent users.”
With all of the figures there, why wouldn’t 343 Industries prepare to support that amount of players on their servers? It doesn’t add up.
Perhaps there isn’t an adequate amount of space on their servers to accommodate that amount of users. But again, surely this can’t be the case. Halo: MCC is a first-party, Xbox One exclusive – published by ‘Microsoft Studios’. Consequently, it has access to Microsoft’s ‘dedicated azure servers’ – remember, the ones servers that they continuously raved about during the Xbox One launch? Yeah, those. We understand that Halo: MCC has to share these servers with other first-party games, but it’s impossible that Microsoft would have run out of sever space to host them all – especially this early into the cycle of the Xbox One. But fine, again, for the sake of this argument, let’s assume that they have run out of server space to accommodate this many players; couldn’t 343 Industries just ‘rent’ out third party servers to host their game for the opening week (when it’s likely to have the highest amount of concurrent users online)? We’re not experts, but we can’t understand why this isn’t a logical solution.
So, it seems unlikely that 343 Industries simply misjudged the amount of users that they’d have during the opening week. So what else could it be?
Ben Cousins also suggested that the broken multiplayer could have been the result of 343 Industries “being cheap”. This view was also echoed by Reviews Editor at polygon.com, Arthur Gies, when we picked his brains on the issue.
This notion suggests that 343 Industries are being too stingy to fork out on additional servers for the opening weeks of the games release and are willing to wait for all of this to blow over. And furthermore, in a couple of weeks time (when the hype has died down and there are less users trying to play the game at the same time), the game will work perfectly and all will be forgotten. But again, surely this can’t be the case. The negative press surrounding the broken servers would radically reduce potential future sales, losing the company millions in potential profit, and thus, it would negate the attempt to ‘save money’ by scrimping on servers.
When we propositioned Ben Cousins with this idea, he replied: “Sales lost from negative press is much more difficult to put in a spreadsheet than money saved from fewer servers.” A fair point, but surely the big-wigs at Microsoft take a more complex financial approach than this – it’s economics 101.
Cutting costs (if you can even call it that) can’t be the reason for games launching with broken multiplayers. After all, Halo: MCC, GTA: V and Diablo 3 are big games with big budgets. For instance, GTA: V had a budget of £170m – bigger than that of the Avengers, Inception and Avatar. Video Games aren’t a subculture anymore, they’re a prominent pillar of the entertainment industry. Why would they want to scrimp on server costs? Patrick Klepek (News Editor at giantbomb.com), agreed with this sentiment – he told us: “Doing online ‘well’ is hard. If Rockstar [the developers of GTA: V], a company with infinite resources can’t pull it off, it says something.”
So if a lack of server preparation isn’t the cause of the problem and neither is the money, what is?
Patrick Klepek told us what he thought caused this trainwreck:
“Halo was probably rushed”
It makes sense. Remember that 20GB patch? There’s your clue. It wasn’t a patch – it was a ‘getaround’. It was a way for the developers to finish developing the game even after it had gone into distribution.
You see, when a game is in production, it has to ‘go gold’ (completely finished and ready to play) quite a while before it’s release date. The reason being: companies have to allow time for the manufacturing and distribution of a game – creating millions of copies of a game onto disc, putting them into boxwork, and shipping them worldwide takes time. So, there is a small window of time (about 3 weeks) where the game has finished its development cycle (‘gone gold’) but hasn’t been released (on store shelves).
However, in the case of Halo: MCC, it obviously hadn’t ‘gone gold’ when it entered the manufacturing and distribution stage.
Presumably, in order to have the game ready for the distribution deadline, the multiplayer was obviously sacrificed. The only way to get around this, was by 343 Industries continuing to work on developing the multiplayer mode during that 3 week window of manufacturing and distribution – hence the mandatory 20GB ‘patch’. You see, it wasn’t a patch, it was literally a chunk of the game that they hadn’t finished in time.
“Have you ever had a boss who made an unreasonable request and when you objected he shouted ‘I DON’T CARE JUST GET IT DONE.’ This is what consumers say to publishers, and this is what publishers say to developers, and these are the results.”
The high consumer demand for the release of Halo: MCC put pressure on the publishers, and in turn, the publishers put pressure on the developers. Consequently, corners were cut and a bad product was made. In the end, no party is happy.
So who’s to blame? The gamers? The big-wigs? The developers? Decker told us:
“I think everybody’s to blame, but the responsibility lies with developers and publishers to fulfill their promises.”
It’s sad. This is what happens when an industry trend of releasing games on a yearly cycle becomes the norm – developers are stretched beyond their limits and sub-par products are released.
And let’s be straight – this article isn’t written by a man-baby who is throwing his toys out of the pram because he can’t play his game. Change the industry of the item – what shall we say? A coat? A camera? A car? This is a product being sold without a prominent feature that was promised. Furthermore, not just a trivial feature – it’s not as if we’re missing an option of having the subtitles in Swahili – this is a core component of the game.
This situation encapsulates the game industry in a nutshell – self-entitled fans, money-driven big-wigs, unreached potential. Embarrassing for all.