Improbable Icon

[RESOLVED] Incident: Google cloud outage impacting US deployment


#1

Due to a Google Cloud networking failure, we were experiencing issues with deployments launched in the US clusters. Impact ranged from clients not being able to join existing deployments to deployment logs and metrics not being available.

We began experiencing issues on Spatial side around 12.14am and the issue has been mitigated now from GC side. We are currently checking whether everything is fine on our side.

We do strongly recommend restarting any US deployments that you had running at the time of the outage. Logs and metrics from this time period are unfortunately unrecoverable.

I will keep you posted and let you know when we can officially confirm that we’re back to full health :muscle:


#3

We have run tests on our platform and all looks fine :tada:

Please let us know about any potentially related issues that you experience! Fingers crossed there are none :slight_smile:


#4

I have issue with my cloud deployment. My project name is beta_spaghetti_seven_891.
I did run it a few days ago with success, but when I tried to run it today it doesn’t start managed workers. And it’s not that they crash, no. It doesn’t attempt to do that at all. Config hasnt changed even a bit since last test we did. I guess it has something to do with recent issues on the google side. Although it doesnt matter if I run EU or US cluster.


#5

Alright! It was my mistake, sorry. My snapshot wasn’t saved properly for some reasons :open_mouth: and read acls of my entities were empty so obviously spatialos wasnt feeling need to run any worker :wink:


#6

Good to hear you’ve found and resolved the problem! ACLs can be a tricky one to spot :slight_smile: