Improbable Icon

SpatialOS Forums

Snapshot is valid locally but not in cloud

I am in the process of upgrading from Unity_GDK 0.2.2 to 0.2.4. I got everything compiling and running locally, built the cloud workers and uploaded them. I then tried to deploy to the cloud with the same default_launch.json and same default.snapshot that worked locally. When I did so, I get an error in the cloud deployment logs saying:

Snapshot did not pass validity check:
Failed reading entity data from snapshot. If this issue persists, the snapshot may either be corrupted, or was generated against an outdated/incompatible schema.

followed by

SpatialOS runtime startup failed.

Is there any reason that a snapshot would be valid locally but invalid in the cloud?

Thanks!

Hi @joeys,

It looks like there could be an incompatibility between the schema.descriptor file and the snapshot that you’re using in the cloud deployment.

Could you please use the menu option SpatialOS > Generate code to rebuild the schema.descriptor file.

Why this was causing varying results in the cloud and local deployments is something we’ll have to investigate further though. In the meantime, please do let me know if your deployments are successful after trying this.

Many thanks,
Ieuan

Hi @Ieuan,

Thanks for the suggestion! I have tried a few things but still no luck. I regenerated the code as you suggested, replaced the entire build directory containing the descriptors, manually deleted the generated .cs files and regenerated them, changed the namespace of schema files that were custom code but still in the “improbable” namespace and removed all but the first entity in our snapshot. Even so, I am still getting the same error. Interestingly, during the course of these changes, I have also intermittently received the “invalid snapshot” error in local deployments, though I am now back to the situation where it functions locally but not in the cloud.

Any info about what causes this error to come up would be greatly appreciated. Also, more detail about the function of the snapshot.descriptor file may help.

Thanks!!

Hi @Ieuan,

I have things working now, but the issue that led to this behaviour implies a possible bug in the logistics of building schema descriptors and running local deployments.

When running a local deployment, the descriptor in build/descriptor/output seems to be copied to build/assembly/schema. When running “Generate Code” a new valid descriptor is written to build/assembly/schema, but not build/descriptor/output. This means that when running a local deployment, the valid build/assembly/schema descriptor is overwritten with the outdated and possibly invalid descriptor from build/descriptor/output. If the build/descriptor directory is removed, sometimes the copying does not take place and the local deployment runs fine; but sometimes running a local deployment causes the build/descriptor directory to be recreated with a descriptor that is not up to date, leading the valid descriptor in build/assembly/schema to be overwritten. (I’m not clear why sometimes that directory is rebuilt and sometimes it is not)

Considering all this, the stable configuration is building a descriptor, then manually copying it from build/assembly/schema into build/descriptor/output so that the build/assembly/schema descriptor won’t be overwritten. This causes both our local and cloud deployments to work; however, this is an easy step to forget.

Does any of this make sense? Any idea if this implies a problem on my end or on yours?

Thanks!

Hey @joeys, thank you for that break down. I think this definitely warrants some further investigation so I’ll see if we can get that done and I’ll let you know if we come up with anything.

Great, thanks!

Hey @joeys, we’ve done some investigation and we believe this is down the the change in 0.2.3 whereby the feature module schema is no longer moved to a global schema folder. Because of this change we no longer wanted the local launch to generate a schema.descriptor (as it would not include all necessary schema) and so we added the flag --enable_pre_run_check=false to the local launch flow.

However, as you still had a schema.descriptor in build/descriptor/outout, the local launch was picking this up and causing the failures you were experiencing.

The solution to this is to run spatial worker clean (or git clean -xdf if you know you’ve checked in all files you need) to reset your project before rebuilding with the new flow.

Note however that this will delete the contents of the build.

Hopefully you’re back in a situation now where your local and cloud flows are as described in the docs.

Please let me know if you have any more issues,
Ieuan

Hi @Ieuan,

Thanks! I have used “Clean all workers” within the Unity GUI a few times without the issue getting fixed, but haven’t used “spatial worker clean” on the command line. I haven’t been cleaning from the command line because, in a possibly unrelated issue, I have been having some issues with “spatial worker *” commands lately. Not sure if this is related, but when I run spatial worker clean, I get the error message:

Running task ‘clean’ for ‘UnityGameLogic’
[1/1] > clean No-op
Encountered an error running command: ‘echo’ with arguments ‘No-op.’: exec: “echo”: executable file not found in %PATH%
Please make sure the step ‘No-op’ defined in file ‘C:\Workspace\Alpha\Prototype\workers\unity\spatialos.UnityGameLogic.worker.json’ is configured correctly.
[1/1] x clean No-op (30ms)
error: exec: “echo”: executable file not found in %PATH%
failed (30ms)
Error during command execution: exec: “echo”: executable file not found in %PATH%
‘spatial worker clean’ failed (0.1s)
Encountered an error during command execution.
exit status 1

I have been mystified by this, because ‘echo’ works fine if I run it manually (running in windows). Unless you have a quick fix for this, I will try the git clean command once I get all my current workspace changes committed.

Thanks again!

Ah sorry @joeys, I think with the GDK for Unity the advised solution is through the git command.

The spatial worker clean command is defined in the worker config files and is template/setup specific.
They are not expected to do anything in the GDK for Unity as it shouldn’t normally need cleaning.

In 0.2.4 we stopped deleting the schema.descriptor in the “Clean all workers” command as it could give you other unintended issues when running a local launch.

If you run git clean -xdf, then “Generate Code” (which creates the schema.descriptor) you should be good to go.

Again, please do let me know if this gives you any issues and apologies for this inconvenience!

Got it. I’ll give this a try and get back to you as soon as I bring my workspace to a stable state.