Improbable Icon

Multi-threading and Spatial


#1

So, since Spatial workers can split the game world to an arbitrary degree, there seems to not be a need to multi-thread in our code at all? ie. if a heavy process such as pathfinding started on a worker, it would calculate a higher load, and would spawn a second worker to handle the pathfinding load (basically cut in half the # of entities each worker is calculating pathfinding for)… in this way, eventually you could have a single worker on each thread of the server hardware you folks are using anyways, so not much to be gained by also using multiple threads in our code?

If this is the case, then instead of designing our code to move “heavy” cpu work such as AI onto separate threads, instead, we should try to encapsulate the work to entities and “controller” entities, while letting Spatial automatically divide up the load by spawning additional workers?


#3

Hi @Swizzlewizzle, the answer to this is a little nuanced, as it really depends on the kind of worker you’re dealing with.

There are two important optimisation concepts to think of:

  1. How inter-dependent your data is
  2. How parallel your computations are

Point (1) is important because we can reduce unnecessary data duplication (why send the same data over the network to two workers to do similar work?).
Point (2) is important because the locations of the bottlenecks in your computation can influence where you want to parallelise.

Let’s take a few examples of kinds of workers you might have in your game:

1. An inventory worker

This worker manages player inventories: the items they’ve picked up and dropped, validates against item duplication issues, and maybe updates some external database so that players can view prices of items offline.
In this case, every item the worker is dealing with is independent of all the other items. Parallelising is simple, because you just handle each item individually.
In this case, imagining a worker as being a ‘thread’ is a perfectly valid assumption, and you can simply start a lot of them to handle your load.

2. A physics worker

This worker simulates a physical region of space. There are several ways you can exploit multiple threads to make your simulation more efficient. However, you do get benefits from having a locality of the world (e.g. all objects within a 1km by 1km box) on the same worker, because you want to avoid density problems. You don’t want workers ‘fighting’ over a region of space, as they all need to synchronise over the network with each other for entities that they share. In this case, having one “heavy” well parallelised worker has an advantage over having several “lighter” workers.

3. A pathfinding worker

This worker finds a route through some obstacles from point A to point B. While calculating any given route is perfectly parallelisable, pathfinding benefits from caching. If you want to make your computations less expensive, you can cache routes (or even better, parts of routes), locally on your worker. This means that having a single worker might have benefits over having several workers, even if each calculation itself is reasonably parallel.

There are some other considerations around how you want to set up your workers’ threading models. Some aspects of gameplay, e.g. physics, benefit from having threading models that keep the simulation smooth (aiming for a constant framerate). This would mean isolating your handling of network I/O, from background tasks, from actually providing frames at a constant rate.


#4

Hmm I see.

Does your backend allow workers to run on their own threads? Ie. I could very easily parallerize my inventory workers by simply having a lot of them, for instance one per character… so with 100 characters, 100 workers each basically having their own “thread” and working 100% in parallel by default.

I’m assuming that this is the way you have designed your workers? As a one-thread-per-worker model? It seems reasonable that if that is the case, positioning and space is used as much as possible to cut game logic up into manageable chunks. For example, instead of having some mammoth worker that all characters need to reference for getting “magic power” of some sort, make it so that regional/local workers can give that “magic power” to the characters by themselves (perhaps with a very low-cost check with a “global” scope worker if required).

This is the way your worker system actually works? I’ve just noticed in the documentation that there isn’t a lot said on actual low-level threading and how the workers are handled… it seems important to know this to, like you said, optimize things.


#5

One-thread-per-worker is not entirely accurate: it is possible to make use of several threads for a single worker.
At the moment, depending on the template you select (e.g. small, medium, large) in your deployment configuration, you will end up with a certain amount of cores to run your workers across. If you start running many more workers than your template can handle, is possible for them to start contending with each other for resources (e.g. if you start more workers than you have cores).
The actual back end details of provisioning resources for workers are something we’re currently reworking. What we hope to be able to provide is more transparency around specifying and isolating resources for workers.


#6

I see. For any serious MMO/simulation environment, of course the per-worker performance specifics are crucial to avoid wasting resources.

I look forward to seeing how things change after the rework. :slight_smile: