Improbable Icon

State saving - How with SpatialOS?



In the other discussion regarding Authentication Options, it occurred to me, how does SpatialOS save state per worker (or does it?)

Take this scenario:
I start up my MMO and it runs on 4 workers. In worker A, I have 100 cattle. Over the next 15 days, in the simulation, they breed an additional 50 cattle. 10 die during this time frame. This leaves me with 100+50-10 = 140 cattle.

What happens if worker A crashes? When the server recovers, do I still have 140 cattle? Or does the state revert back to 100 cattle?

I would hope that machine state is saved somewhere. If not, how does something like Worlds Adrift handle this? Are they streaming pieces of blown-up ships to an external database somewhere? If so, how are they seeding workers when they come back alive?

Deployment authentication options
Thinking about SpatialOS for a project but some concerns
Player persistence between sessions

Sooo… jumping over from your other question on authentication.

!! Warning !! Looooonng technical / meta-physical piece of description / rambling coming on. Thou be warned!

There is an important difference to make between the “data” that makes up the state of your (game) world and the “data” that keeps track of the state of your (game) world. I know and admit that just writing that phrase does not necessarily clear things up (some might be more confused even… you have my sincere apologies), so I’ll try to illustrate & explain this as best as I can:

Data that makes up the state of your world

This can include, in a clearly non-exhaustive manner, things like the position of entities, player’s health, etc. This covers things as immaterial and global like the current weather in a region of your world to physical and local things like the content of a player’s inventory (I deliberately choose that last example). If you relate this directly to SpatialOS notions this basically targets all the components that you describe in you Schema files, multiplied by the number of entities that use each specific component. In sum… this is a very diverse set of data.

One thing though that characterises this set is that each element in it is poised to change quickly and continuously throughout the life-cycle of the deployment where they reside. Yet another, perhaps more fundamental, point is that without this particular set of data there is no world: this data is the world.

As a consequence of the two previous common characteristics this kind of data needs to be available quickly and at all times, potentially for all the workers running throughout your entire deployment running on multiple machines (which is after all the raison d’être of SpatialOS). This means this data is stored in memory (i.e RAM) distributed across the different nodes / servers of your deployment. This is done, not in the workers themselves but rather in the core part of our platform that also performs load-balancing, etc. Workers can fail, time-out, start, stop, pause without affecting the data-store. When a worker sends & receives component updates it’s talking to this inner layer of SpatialOS and not directly with other workers. On start a worker just receives a larger set of updates than usual as it does a full check-out of the components / entities in which it is interested.

When SpatialOS promises persistence it should be understood as in: "the data that makes up the state of your world will be available from the start of your deployment to its stop no matter what happens to the workers that simulate your world. Still, even if we implement multiple techniques to ensure that this data storage survives when all hell breaks loose, we are not sheltered from a potential major outage (e.g a cloud provider messing up the data-centre in which your deployment is running). To this effect, and for numerous other use-cases, there is the possibility to make so-called snapshots of your world which are basically a dump of the whole state of the world at a given point in time.

Such a snapshot can be used to boot a fresh deployment in the exact state of the old deployment when the snapshot was taken. This is another part of the persistence of your world that SpatialOS is all about: the entire state of your world can even survive death of the underlying deployment.

Now you may put forward that you also don’t loose your position, inventory and attributes when your favourite MMORPG’s servers go down for maintenance or crash but such worlds do not retain all data (the position and health of this NPC, the fact that the tree at location < X,Y,Z > has been chopped down).

For a more concrete illustration: most NPCs in such games are based on the well-known spawn cycle which mean that you don’t care about the state of any particular NPC before & after a server reboot. And… well… with SpatialOS we do care. :stuck_out_tongue_closed_eyes: This is one of the key elements with which we want to encourage people to play in order to invent completely new game mechanics and types. But now… back to snapshots…

Snapshots are a costly operation and should not be counted on to offer the possibility to, for example, roll back exact player transactions with respect to maintenance or other similar incidents. For similar reasons it is also not a very good design to store all of your data related to offline players somewhere a Schema in your world: it will only increase the size of the snapshots and the cost of making them without providing the proper guarantees for rollbacks. This type of guarantee should be provided with…

Data that keeps track of the state of your world

Not all data is born equal and there is no Universal Declaration of Data Rights. As a developer of an online game you are probably much more attached to the data that describes the attributes and belongings of your user’s characters than you are attached to the exact position of that chopped-down tree at < X,Y,Z >.

In a general manner, even outside of SpatialOS, as soon as you have such an element of data about which you care deeply you want something that:

  1. is written on disk and not in memory, preferably on multiple disks in multiple locations for redundancy.
  2. has access mechanisms that implement transactions correctly.
  3. (potentially) allows for rollback to a specific point in history.

… in other words an on-disk cloud-based transactional database.

This does not mean that the same elements can not also exist as temporary duplicates in your deployment as pieces of data that make up your world but you do want to back those temporary storages with a permanent one. The size of this should not be enormous even for huge deployments as the amount of data that you really care about is only a very small percentage of the total volume of the whole world state.

SpatialOS in not an on-disk database as I hope the first part of this waaaay too long post manages to explain. Maybe… in the future (no promises)… we may want to (no promises, again) to investigate such extensions. However in the meanwhile there is still a lot of fine-tuning to do on the current scope of SpatialOS and we do want to put our efforts where we believe that we can provide the greatest value to you as a developer. This means providing a platform that allows for small teams and even individual developers to very rapidly create and iterate on new innovative games that grow to the scale of what AAA studios do with a full network and infrastructure engineering team, or beyond.

Anyway… I’m signing off here (and if somebody knows the address for a rehab facility that treats people that write too long response posts, please forward :laughing: ).

[BEST PRACTICE?] Character's change effected by time and the system: how/where to store?
Some questions about external connection

Understood. I am talking of the non-static entities as it relates to maintaining state. And correct again, as I am talking about non-memory persistence as that is what I am most concerned with in an evolving world such as allowed by SpatialOS.

So to your answer, “SpatialOS in not an on-disk database”. This tells me that when I do indeed spin down my world (for maintenance or update or whatever,) that trees cut down at (x,y,z) are no longer cut-down. In fact, my world will resort to the snapshot I created when I last built it. In reality, all of my entities and their evolved states, will be lost.

Assuming I am understanding this correctly. Is there an API call or method I can use to save the state of the world and update the stored snapshot so that when I reboot the world, it comes back with some persistence? Currently I save my snapshots prior to launching. Can you create a snapshot while launched?

Hopefully I’m making sense.



I’ve been wondering this myself too, especially since loads of things in my game (like chopped down trees, sunken ships, building houses and even cities, states of houses, decay of items, orders in shops that have been made by players, etc) are a vital part in my game for both the main storyline and the experience for the player.
I understand World’s adrift makes it possible to salvage from crashed ships. Assuming they too will have regular maintenance I assume there is a way to take snapshot before a reboot, but I haven’t seen that option yet, or have interpreted things wrongly :smile:

Thanks again to everyone of the Improbable staff and users who are so responsive and helpful :smiley:


From the absence of a counter next to my snapshot link in my first response I can tell that you have probably missed it. :wink:

The documentation page that it links will tell you all you need about snapshots (and here’s the same link again :slight_smile:)



@dvanamst my first thought would be to create a C# worker that acts as a bridge between your other workers and a database solution (I use Playfab for example) so that secrets and passwords are not included in the Unity builds (since the Client and UnityWorker share the same codebase).

Would you also choose such a pattern?

Also: is it possible to send Commands from a third-party provider, such as Firebase or Playfab, into SpatialOS so that you can have changes in your external cloud provider be synced back into SpatialOS?


@draconigra From my perspective that would definitely be a valid pattern although there might be some input @andrew.roper can give about how they handle their exchanges with Firebase in Lazarus.

As to Commands from external providers that is not supported as they would need to be identified in the shape of a worker. You could however use your secondary C# worker as a unique access point that manipulates both the database and can send commands to the deployment. That way for the external changes to take effect you just apply them using the worker as proxy which will ensure that the database gets updated at the same time as commands are being sent.


@dvanamst so you mean that to sync external changes into SpatialOS it is best to use a polling model instead of a push model?


That is one possibility, but it’s viability will of course depend on the frequency of external changes.

If you use my last suggestion about using the bridge C# worker as your entry point for your database then you won’t need polling. In more detail (I know it might look a little bit convoluted but it’s not actually that complicated to set up) you have the following:

  • Deployment -> A
  • Third-party database -> B
  • C# client / external database bridge worker -> C

Whenever you want to make modification to the database you go through C which, being connected to A as a worker can both send the necessary commands to A and send the necessary commands to B’s API. In the case of regular and/or automated changes you can run C on your own server.

If you are in the situation that you have changes to B that you do not control but still want to have reflected in A then I’m pretty sure that B provides an API to get notified of such changes. If not you can still use the same callbacks / event-hooks that you wanted to use for sending commands to A to create your own notification system. The API or your own system would then need to notify C of the change and C would in turn send the necessary commands to other workers in A.

Some questions about external connection

Do I understand you correctly that it would be possible to run a Worker on your own piece of infrastructure instead of inside the Spatial OS cloud? Because I am having trouble seeing how B could message C that an update has been done


Yes of course. :slight_smile:

After all your game clients are just that: workers running outside of the deployment but connected to it. This is not a functionality reserved to Unity clients. It is embedded as a core-principle into SpatialOS. You just need to go through the worker::Locator class in the C# / C++ or Java SDK instead of through the worker::Connection class to get your connection set up. I’m explaining this in one of my replies in the authentication thread.


Got it, I see you can take a snapshot of a running instance. Where is it stored when you do this? Does it try to save it across the internet?

Is there a way to create worker code that is not deployed as part of the client package? I’m thinking about API keys here that I don’t want visible via a decompiler.


Each worker has its own code / binary. So as @draconigra suggested above: