It was a misclick that started one of the largest battles in Eve’s history. A Titan pilot attempting to intervene with his fleet in a far smaller conflict instead drifted out alone into a diplomatically delicate situation. Military escalation did its freight train thing, and soon enough 3000 pilots in terribly expensive ships were slugging it out in lowsec Asakai space. Meanwhile, developers CCP looked on in awe. But how did they keep their servers from snapping?
“Asakai popped off without much warning because even the littlest action can have repercussions for thousands in EVE,” wrote CCP Manifest in a new devblog. “It’s one of the first big lessons people learn about the universe—each decision has a consequence (usually handed out by another player, which is why we named our last expansion Retribution).
“In this case, literally a single botched ship command. A butterfly wing flutters a monsoon. A lone droplet of blood invokes a feeding frenzy. Part of the reason why EVE players love stuff like this story is because they can, at some level, imagine themselves accidentally doing it.”
Now, on Tuesday, it’s a wonderful anecdote – but it was CCP on Saturday who had to tie down the sails and ensure that Eve survived the storm of lag that comes when thousands of players flood to one location.
The Icelandic developers have a cluster status webpage which gleams with big red numbers when a node is overloaded by too many fleets in one place. From there, CCP Manifest explains, the team have a few options:
- We do nothing.Either we expect the fight to be short or the load is manageable. A large majority of fights fall here thanks to our hardware investment. Yay Technology!
- We move other solar systems on the node away from the fight. This disconnects anyone in those systems temporarily, but spares them from the ongoing symptoms of being on an overloaded server. It helps the fight system a little bit as well, especially if a reinforcement fleet is traveling through those other systems.This was done for the Asakai fight over the weekend, but is rare.
- We move the fight system onto a “supernode”.We’ve got a couple machines that are crazy-good hardware, well above what the rank and file of the cluster is. This is the machine that systems get reinforced on when players request that for a preplanned fight of this magnitude.Unfortunately, the same thing above applies – anyone in the system when the move happens gets disconnected.Because of this, it’s basically never done for a battle already in session.In this case, the fight broke out because of a Titan put out of position by accident. Had we gone this route and moved the system, the Titan and his friends simply wouldn’t have logged back in, killing the fight.So, yea, this just isn’t done.
On occasions like the Asakai incident, CCP actually slow down the game’s speed by about 10% to deal with the ludicrous number of calculations necessary.
“If we didn’t do that, the sheer amount of data traveling back and forth might mean one side’s modules and commands wouldn’t go through and all of a sudden it is just a game of flip-the-coin,” writes CCP Manifest.
“Since the system itself slows down but others around the server do not, it actually also helped reinforcements arrive from non-dilated nodes quicker.”
Amazing. Oh, and for the record, Asakai wasn’t the biggest battle in Eve history. That honour still belongs to a scuffle in LXQ2-T.