Improbable Icon

Decentralized Deployments


#1

Hi!

Would it be possible to have workers of the same deployment running on servers all around the world?

What if lag-sensitive logic would be executed on workers physically closer to clients to reduce lag. Of course inter-worker lag would increase but maybe the user experience would be better. The other thing is how to match client to workers that are closer to each others?

Not sure it would be a good thing but has it been tried before? If yes what’s the result?

I wonder how a physic test, like this, would react to inter-worker latency increase.


#3

Hello @SionoiS,

Good question you ask there. Definitely one that must have crossed the minds of many people (including myself) at one point or another.

The idea of moving a physics worker (for example) closer to the clients for which it manages the characters might definitely make sense from the perspective of an immediate benefit to the client. However things are, unfortunately, not that easy and you might actually, counter-intuitively, reduce the overall experience a player has.

A core element in all of this is the very latency that you are mentioning yourself. Let’s take a simple but effective example where we would use such a decentralised deployment strategy:

  • A, B are clients / players located in the same geographical region (e.g Central-Europe) but your game also has player C in USA-West.
  • The deployment uses 2 nodes (i.e cloud instances): 1 & 2 which are located respectively in Europe and the US.
  • The deployment runs many workers amongst which X, Y and Z are physics workers.

Now let suppose that your game is the average MMORPG with a focus around open-world raids because SpatialOS encourages you to have vast worlds where there is no need for instancing, etc.

Initially A, B and C all play their own quests in different parts of the world and are actually managed by different physics workers (A -> X, B -> Y, C -> Z). All good you might say and you could put X and Y on node 1 and Z on 2.

Now it happens that A, B and C are friends and want to play together and join the same raid. This means they will end up in the same part of the world which will be managed by the same physics workers (let’s assume X). Now this means that while A and B will be playing fine with a close-by physics worker but C would suffer from a cross-Atlantic round-trip latency. The same would happen if they end up on Z which runs in the USA, with A and B suffering from the significant round-trip.

Even more interesting would be when the players move across the world together and they move from X to Z (or vice-versa). At that point their latency would jump and the boundary area in the world where the authority regions of X and Z meet will be not a nice place to play. X and Z themselves would significantly suffer from the lag between the two of them without even taking the players into consideration.

For all the above reasons, as well as for many more, there are considerable advantages in keeping all nodes of a deployment in the same datacenter / availability region. Or… to put it differently:

SpatialOS makes it far more easy to scale up your worlds without bothering too much about network-code but it does not however abolish the speed of light.

Best regards,
Duco


#4

Damn you speed of light, can’t you be faster! (shake fist)