Improbable Icon

Deep Dive: Worker Authority

deep-dive

#1

Hello devs,

After our first post in a new series of How To’s that dealt with position and movement optimisation, today we also start a second series of Deep Dive’s where we look into the innards of SpatialOS and try our best to explain how some specific mechanics work and how you can best understand them yourself. Many thanks to @samcorcoran for doing this great write-up!

The accompanying code for this post can be found on GitHub!

Do you like these pieces of content? Do you have ideas for things that you would like to see appear in the How To’s and Deep Dive’s? Do tell us both below in this thread and on the Discord chat.


Introduction

Worker authority is a central concept for working with SpatialOS. In your game you may have multiple types of workers operating on different aspects of your world, and for each worker type you may have many worker instances. Every component of every entity can individually be written to or updated by one single worker at a time: the authoritative worker. There are however multiple reasons why exactly authority over a component may be moved between workers at runtime.

@DavidR , one of our engineers in the Production team, put together a small “tank warfare” demo, with some simple behaviour and tools for visualising worker authority over particular components. We hope that it will help you to better understand how to reason about component authority and it is a great starting point to dig even deeper into this topic by yourself.

Setup

To run the project simply download the repository and run spatial worker build from within the project root directory. You will need Unity Engine installed so if you haven’t already run through the SpatialOS setup for Windows yet you should do so now.

When you start the demo (spatial local launch) the UnityWorker instances and C# worker instances will be started automatically. You can then open the UnityClient worker in your Unity Engine editor and connect to the simulation by hitting play.

The UI within the demo should guide you the rest of the way, but for some more information read on!

Explanation

The SpatialOS load-balancer may decide to re-allocate a component to a different worker, i.e. change authority, for a few different reasons:

  • The original worker crashed
  • Movement of the tank unit to which the component belongs
  • The “centre of gravity” of the workers moved (changing the location of the implicit boundary or overlap between the AI workers)

In each of these situations the load-balancer may decide that the best ongoing authority strategy is one which involves an immediate change. Both the former and the receiving worker are notified of the authority change with an AuthorityChange operation.

Working with SpatialOS, and debugging your logic, requires an understanding of when these authority changes may be occurring in your game. You should then put some thought into how you can ensure that the worker that receives authority over the component is given all information necessary to seamlessly continue the simulation.

This demo project offers an interactive way to visualize authority changes within a toy example. The code is not an example of “best practice” but it should nonetheless be of interest to anyone interested in what load-balancing configurations for multiple worker types can look like as well as for anyone interested in seeing a C# worker and Unity SDK worker operating together in the same project.

The scenario is a tank warfare game in which tanks drive in a circle around the centre of the world. There are two ‘aspects of the world’ to be simulated, each with an associated SpatialOS component:

  • The position of each tank unit (Position); the tank will remain stationary unless the authoritative worker updates the position property
  • The rotation angle of the gun turret (TurretInfo); the turret will remain at a fixed orientation unless the authoritative worker updates the rotation property

Each aspect of the world has a unique worker type responsible for it. Authority for the tank Position component is only ever given to a worker of the type that is built using the Unity SDK. Authority for the TurretInfo component is only ever given to a worker of the type built using the C# SDK. The demo is configured to have four UnityWorker instances and two CSharpWorker instances.

You can find more information about authority, and how to handle changes of authority in the docs.


#2

I was wondering: what are common pitfalls that you need to look out for when authority changes? Based on the contents it is rather clear to me why the authority changes but not exactly what the implications are?

Can you give an example of where an authority switch would cause a bug in your simulation and how to counter that?


#3

Hi @draconigra.

This project is just a tool to visualize how the authority changes happen, so there is nothing really important going on that could be an issue when the authority changes.

For an example of why you have to take into account authority changes when writing your code, imagine that you have some condition-based (clicking, pressing a button or game state related) important mechanic for your game (shooting, activating some device …) and you can’t afford to miss it (as opposed to this demo in where the position is updated and sent continuously),

In that case, an issue that could happen is that you have a worker with authority over a component on an entity when the triggering condition happens, so it sends an event/command for the mechanic, but at the same time, SpatialOS has decided to change the authority of that entity to another worker, so when it receives the event/command from the “previously authoritative worker”, it rejects it sending back a “you don’t have authority” error. You now need to somehow let the “new authoritative worker” know that the triggering condition happened and it now needs to send the event/command.

Another typical example of a possible issue is when a worker assumes that it has authority over some component in a group of entities and it loses authority over one of them. Imagine a vehicle entity and a character entity. The worker responsible for the movement has code that say that when the character is inside the vehicle, it should update the position of the character to be the same of the vehicle (plus an offset depending on where the character is), but when it loses authority over the character, it can no longer update his position, and the worker that got the authority over the character can only update the position after receiving an update containing the new position of the vehicle, which introduces delay in the update and will cause the character to jiggle a bit, potentially causing issues if physics are involved.

One solution for cases like this is to somehow “merge” the entities into one SpatialOS entity and write code that takes into account the union/separation of the group. In this example, when the character enters a vehicle, you could “hide” the vehicle entity in game (physically and visually) and modify the character entity to represent it is in a vehicle and you update the character position normally and make the vehicle entity update accordingly (but since it is “hidden”, it doesn’t matter that there is a delay because the authority over it’s position is in another worker). Or if the vehicle entity doesn’t carry any meaningful data that you need to keep track, you could destroy it completely and recreate it when the the character leaves the vehicle.