Improbable Icon

Deployment Outage 2nd November 2016 [12:20 pm]

announcement

#1

Update: Fully restored

New deployments are failing across all clusters.
Incident Management has been informed and a rollback is being initiated.
Current deployments are not affected.
Next update at 1pm.


#2

#3

A tentative root cause has been identified. Mitigation is undergoing tests and verification.
Both eu1-prod and eu2-prod remain drained for creation of new Deployments.
Next update in an hour.


#4

Scratch that; engineering worked super fast :smiley:

The fix has been verified on eu1-prod, and that cluster is and ready for use.
Other prod clusters (eu2-prod, us1-prod, etc) remain drained. Work on applying the rollback fixes and validation commences.


#5

Fix verified and rolled out across all clusters.


#6