The studio goes into lockdown. The lights dim. Programmers frantically type code and run between clusters of desks. Silent warning lights pulse in the bright server room as coveralled engineers slide out slabs of silicon and perform delicate operations on them, wiping beads of sweat from their brows.

Then, with seconds to go, everyone finally completes their tasks, and the Chief Engineer pushes a large lever back to the ONLINE position. The lights flicker and everyone catches their breath, waiting for the server’s computerised voice to confirm maintenance has been successful. Then everyone cheers. The May update is complete. Well done, everybody.

That’s the scene I imagine when a game goes into maintenance. I suppose I need to feel there’s the kind of high stakes drama going on in some distant office or server farm that justifies the fact I can’t play the game I want to. And obviously, it’s not what really happens. Keeping online services for big games running is a slick and controlled business, honed through years of experience and best practices, and necessitated by the expectations of thousands or millions of players and millions of dollars of investment. 

"In theory, the number of things that could go catastrophically wrong is rather terrifying," says Glen Miner, technical director of Warframe. But in the face of all that threat, developers work hard to minimise it. "The last update we deployed only took 26 seconds."

Warframe is subjected to regular weekly updates, but new features and fixes are added as soon as they’re completed, so overall, the game is updated several times a week, and sometimes several times a day, in a process that’s honed so sharply that it’s almost always completed within two minutes.

"The most common thing we do is upgrade the server software to match updates to the game," says Miner. "This involves pushing server code, restarting scripts that keep the world alive, and enabling new content." Digital Extremes aims to give players new stuff as soon as possible, so the bulk of its maintenance is about doing little changes as soon as they’re tested and ready. 

Less frequently, maintenance is about hardware. "Particularly SSDs," says Miner. "A few years ago we had server problems right before Christmas that we traced back to an SSD that had become exhausted by the firehose we had been blasting it with. We had to perform some emergency upgrades while the holiday load got heavier and heavier which was extremely stressful."

For Worlds Adrift, Bossa Studios’ physics sandbox MMO in which players sail airships and swing around on grapple hooks, maintenance is carried out daily, and it’s all about preventing the world from getting out of control. 

Most MMOs avoid physics and having persistent objects rolling around, because they’re incredibly difficult to govern over a network. But Worlds Adrift is not your usual MMO.

"Everything players do in the world is remembered by the game," says lead developer Tristan Cartledge. "If a player cuts down a tree or destroys a ship on an island, the remnants of their actions will persist in the world until another player or a natural phenomena, like a storm, comes to disturb that state. Because we are storing all this information, the size of the data required to record this can grow unbounded."

The longer the game runs, the more memory its servers require to keep it going, and so Worlds Adrift’s regular maintenance is all about taking the game offline for an hour and cleaning up a snapshot of the world and running compression algorithms on its data to reduce and remove anything that isn’t important. Perhaps the game doesn’t really need to remember the exact position and rotation of a Thuntomite’s corpse or the amount of wood left in a log, and can estimate it instead. But important objects, such as ships, chests and living creatures, are left completely intact.

This process is now entirely automatic, even down to Worlds Adrift’s system flying bots out into the world to test things, ensuring that its physics is active by cutting down a tree and other checks. In fact, the development team wouldn’t know anything about what’s going on unless the bot spots something’s up and sends out an alert. 

Not that Bossa hasn’t experienced some weird problems. A while back, Worlds Adrift had a bug in which the fuel pod item wasn’t spawning into the world correctly. While they worked on a proper fix, the team spawned them manually during maintenance but failed to take into account the fact they wouldn’t all be harvested between one day and the next. As the days passed, the number of fuel pods in the world after maintenance grew and grew until they had hundreds of them on each floating island. "It made them look very much like strange sorts of hedgehog," says Cartledge.

"The only stress comes from a low-level worry that something will go wrong during maintenance which could result in a snapshot being corrupted." In that case, the team will have to roll back the world to the last good shapshot, which could be between 10 minutes and a whole day of lost progress, depending on what happened. Not ideal.

For Digital Extremes, updating Warframe is similarly stress-free, aside from dealing with dead hard drives at Christmas. "The most stress comes from problems that are outside our control," says Miner, remembering situations in which the whole game was at the mercy of network issues affecting their own suppliers. "In cases like that we’re practically helpless and it’s extremely stressful."

Fatshark, maker of the Vermintide series, have offloaded the stress of maintenance entirely. For the first Vermintide game, they built their own backend platform, which was regularly maintained. "That took quite some effort from our IT team," says CEO Martin Wahlund. So for Vermintide 2, they turned to a third party company called Playfab to take care of all the game’s online services so Fatshark can focus on development.

Playfab even performs maintenance without needing to take the game offline, so Fatshark doesn’t have to worry about keeping players abreast of day-to-day fixes.

Digital Extremes is also able to do live updates with most of Warframe’s maintenance. Some updates flow out to its datacenters ahead of release so they’re all ready for when the team flips the switch. Many updates simply happen in the background, with the only effect on players being that they can’t save until they’re complete.

A minority of software or hardware upgrades might require the game to be taken offline, but even here, players can keep playing. Since Warframe runs on clusters of servers, the team can take a node out of service, tend to it, and then add it back into the pool without players noticing. 

Given that these tasks reduce the capacity of the system, Digital Extremes schedules them for times of the day when there’s less activity. Bossa schedules its regular maintenance in the same way, depending on whether the servers are in the Europe or the US. "We attempt to do it as close as possible to off-peak but we still have to run maintenance during office hours for Bossa, so there are team members available to intervene if anything goes wrong."

Bossa schedules its updates around staff availability, too, particularly QA, who are there to check that everything runs correctly for when the game goes live again. They can’t practically perform rigorous testing because it would take too long, but they can ensure Worlds Adrift’s most basic features still function, like physics, ship building, flying and character progression. 

Naturally, QA will have already tested all of a game’s new features prior to release, so the period before maintenance is usually more fevered than maintenance itself. That’s certainly true for Digital Extremes. "Since we’re always trying to cram as many improvements as we can into each update, there’s usually a frantic sprint of ‘just one more change, please,'" says Miner. 

"When we start the countdown and start running the scripts to make the changes, there’s a brief window of terrified calm while we wait to see if we missed anything," he continues. The maintenance script resets a leaderboard which details all Warframes crashes, and the developers’ eyes lock on to it to see if the bugs they fixed stop appearing on it.

Then the community team fires up. "No matter how big your QA team is, your playerbase is usually thousands of times bigger and players can often be extremely helpful," says Miner. "Sometimes the most rare and unusual bugs can be fixed easily when community managers can get us diagnostics from players and so they’re often busy after an update, collecting and isolating problems the players have found."

But the real challenge isn’t so much the maintenance, nor even checking that it worked. Maintenance, paradoxically, is often the calm before the storm.

"The main issues with maintenance for Vermintide 1 have been when it was over and a lot of people tried to login to the game at the same time," says Wahlund.

It’s the same for Warframe. "One of the things that’s been a regular challenge is dealing with an ever-increasing number of players hammering our servers waiting for the maintenance to be over," says Miner. Even though the team optimised downtime to just a few minutes, the sheer volume of network connections in that time was enough to overwhelm Digital Extremes’s systems. 

"Several times this year this stampede was even bad enough to trigger problems with our network partners. Luckily, we were able to upgrade a key network device and, with some clever configuration tricks, we’ve managed to practically eliminate this problem for now."

Maintenance is necessary, complex and dangerous. And that’s just the kind of challenge that inspires a company to work to make it as painless as possible for players—and for themselves. There’s a lot that’s magic about how games connect players and let them play together, but updating and fixing themselves while they’re still running has to be one of their cleverest tricks.

I played Vermintide for a couple of years without mods and liked it plenty, but when I finally installed a few (via this modpack) they immediately became essential. With better bots and the ability to drop the HUD at the press of a button, I can turn it Vermintide into a game about photographing the Old World. A squad of competent AIs protects me while I wander around setting up perfect angles on Tudor houses and twin moons, something no human players would ever put up with.

At the other end of the spectrum are mods like the Stormvermin Mutation, which upgrades enemies so that what would have been slaverats become clanrats, clanrats become armored stormvermin, and every special is replaced by a rat ogre. The first time I tried that I wound up facing two ogres and a pack of stormvermin simultaneously on a narrow bridge. It didn't last long. 

And there are plenty of other mods available for the original game, many in the same collection. There's another difficulty increase called Onslaught, sound replacers, UI tweaks, cosmetic changes, and even a separate mod that adds a playable chess set. Because why not?

Official mod support was planned for the original Vermintide but is yet to appear—it's still there, grayed-out and "Coming Soon!" on the launcher's menu. Fatshark's technical director Robin Hagblom tells me it's still coming, "but Vermintide 2 is top priority at the moment. Hopefully we'll open both workshops simultaneously, but no promises."

The initial lack of official support didn't dissuade modders, of course. According to Grimalackt, who maintains the Quality of Life modpack, it began with cheats. "Modding in vermintide 1 started as a DLL injection," he says. "Its first uses were very clear-cut cheats. 'Win game' buttons. Adding all the items to your inventory, god mode, etc." 

As he explains, they were designed to mess with the code in the middle of a game. "The possibilities given by that are obviously beyond game-breaking, especially since a lot of the game was at first coded with the assumption that the other clients were running exactly the same code as you were." He gives examples of potential malicious uses like kicking hosts out of their own games, impersonating other players in chat, or crashing someone else's game without even being in the same lobby as them.

The mods that became popular weren't the ones that let you cheat or troll other players, however. "I think the biggest reception was to the various sound replacers I made," says modder Aussiemon. "This was one of the types of mods made possible with Fatshark's assistance, as nothing like it had ever been done before. I wrote a script to play the 'headshot' sound from Unreal Tournament 2k whenever a special skaven was shot in the head." People started sending him requests for things like the MLG airhorn playing when gutter runners attack and soon he became the go-to guy for all your audio mod requests. "It ended with John Cena's theme music playing when an ogre showed up, and that silly balloon hammer mod." 

Nowadays the most popular mod is Grimalackt's Quality of Life  collection, with its toggleable options like third-person camera, the Stormvermin Mutation, and those bot improvements I l like so much. Some of these things were made possible by Fatshark giving modders access to their SDK tools, creating a Steam community for them, and answering questions directly. "We'd post about an issue we were having, or access to an uncompiled file for reference, and Robin would reply within the day with a suggestion or reference file. These discussions led to the first custom models, textures, and sounds in Vermintide 1, and we definitely would never have gotten that far without Fatshark's insight.

"It wasn't just their advice though," Aussiemon continues. "Fatshark gathered our suggestions and thoughts on mod support to form a plan for Vermintide 2. I think a not-so-insignificant part of the official mod design will be the result of a collaborative effort between Fatshark and the modding community."

What's next?

Fatshark is aiming to add Steam Workshop support to Vermintide 2 in late April. "Though even past this it will still be an ongoing project," Hagblom says, "improving the tools, adding more functionality and adding access to more parts of the game so it won't be a fire-and-forget release."

They've made a Discord channel to connect modders and share information, working with them to map out how to best implement it. "It's really paid off for us too even without the actual mods," says Hagblom, "because they've in turn helped us finding repro cases for bugs and reported when they've found things that haven't exactly been working as intended."

Bringing everyone together before mod support goes live means that when the Steam Workshop goes live there will be plenty ready to fill it. "The first priority will obviously be to transfer everything from Vermintide 1 that is still usable in Vermintide 2's context," says Grimalackt. "Since Vermintide 2 still shares a lot of code from the first game, the task is usually not too difficult, with some mods sometimes working after a simple copy/paste with little to no adjustments. Third-person for example falls in that bag. It was already made functional in Vermintide 2 within days of one of its betas, although obviously can't be shared yet until official support comes around."

As well as third-person mode he says visible damage numbers and health bars, chat blocking, bot improvements and crosshair customizations mods will all make the leap to Vermintide 2. Aussiemon is already working to bring over Fashion Patrol, which turns stormvermin white, as well as his mods for skipping cutscenes, freeing up hat cosmetics so anyone can wear them, scaling the UI for 4k, and adding more mission stats.

Following the transfer of old mods, Grimalackt predicts UI improvements will be the focus. "Any of the complaints about any of the UI screens that Fatshark doesn't fix themselves, and that are within modding powers are going to get fixed by mods." 

Aussiemon agrees. "Some people want detailed weapon stats. Some people want faster endgame screens. Some people want us to port over the UI from Vermintide 1. Working with the UI code isn't easy, but this will probably be what people will focus on first anyway. UnShame is already working on a 'weapon stats' tab at the equipment screen, in fact. I'm sure Fatshark will eventually implement some of the common suggestions themselves, but there'll always be ideas for modders."

And then, of course, the stranger ideas will flow in. Aussiemon says that after texture, model, and sound replacers, "eventually the Vermintide 2 tools will let us make new items, animations, cutscenes, and even levels." In the first game he managed to turn the Pub Brawl, a limited-time event that let players biff each other up in the Red Moon Inn, into something players could enjoy whenever they wanted. Now he's working on a follow-up.

"I've written a mod that allows friendly fire in Taal's Horn Keep," he says, going on to explain that without access to bespoke animations like those the first game had for Pub Brawl, like fists and Lohner pouring flagons of Bugman's ale, it won't be exactly the same—unless Fatshark does something like Pub Brawl for Vermintide 2. "Pub Brawl was a fan favorite though, so I bet we'll see it make a comeback. When it does, I'll have a mod."

A barricade against cheats and trolls

Modding Vermintide 2 won't be a complete free-for-all, of course. Some players will want to hog-wild while others would rather stick with the vanilla game. "To cater to both these things we will be splitting the game into two realms," Hagblom says, "the official realm and the modded realm. So if a player is playing in the official realm, they will be able to know that everybody else playing are playing legitimately. Though, since we've had a lot of mods for Vermintide 1 with QoL improvements, more advanced UIs and the like we still want this to remain without forcing these players to play in the wild west of the modded realm. To solve this we will allow mod creators to apply to get their mods vetted for 'sanctioned' status. We will then go through the mod, make sure it doesn't contain any cheats or unfair advantages and if everything checks out, clear it to be played even in the official realm."

As Grimalackt says, "Those who want to create absolutely ridiculous mods will now also be able to share them more openly, since there will be an 'untrusted' realm with separate matchmaking where everything is permitted, at the cost of loot not carrying over to the trusted realm."

Something we probably won't see is rebalancing to make the game easier, much as I'd like it if mid-level bosses like Bile Trolls and Chaos Spawn had about half as many hit points on Recruit difficulty. Grimalack calls it "very unlikely" that custom difficulty rebalancing will make it through the new system. "The bot improvements will likely still make a comeback, however," he says. "The only complaint the devs ever really had about them was their sometimes ridiculously accurate aim, so that part might have to be left unimproved, but their handling of tomes and grimoires will almost assuredly be changed just like in the first game."

As for making the game harder, "Vermintide 2 is already significantly harder than the first game. Some of the heroic deeds are also heavily inspired by the first game's difficulty mods. The 'Vanguard' modifier, is effectively what Stormvermin Mutation was, as it directly turns clan rats into stormvermins, and slave rats into clan rats (and does the same to Chaos equivalents), just like the Stormvermin Mutation once did. It doesn't turn all specials into bosses, however, so I guess I still have something to do.

Lines will be drawn at mods that infringe on intellectual property. "So for instance no Lord of the Rings mods," Hagblom says. "When the Mod SDK is released it will have a EULA specifying the exact dos and don'ts."

Grimalackt is looking forward to the fact he won't need to maintain the equivalent of his Quality of Life modpack for Vermintide 2. "Every modder will be responsible for uploading and maintaining their own mods, and won't have to go through me to have me include their work in my package. I'll be able to dedicate more time to creating my own mods instead. Fear not though, the QoL modpack will eventually still live on as a Steam Workshop list of recommended good starter mods, sometime down the line."

He already has ideas for what he'll do with the free time he'll have. "One thing I also definitely have to do sometimes soon is turn everything into plague monks. You don't see enough of those guys around. I can already imagine the screeching from a horde of monks…"