Improbable Icon


Deployment Outage 2nd November 2016 [12:20 pm]



Update: Fully restored

New deployments are failing across all clusters.
Incident Management has been informed and a rollback is being initiated.
Current deployments are not affected.
Next update at 1pm.

pinned #2


A tentative root cause has been identified. Mitigation is undergoing tests and verification.
Both eu1-prod and eu2-prod remain drained for creation of new Deployments.
Next update in an hour.


Scratch that; engineering worked super fast :smiley:

The fix has been verified on eu1-prod, and that cluster is and ready for use.
Other prod clusters (eu2-prod, us1-prod, etc) remain drained. Work on applying the rollback fixes and validation commences.


Fix verified and rolled out across all clusters.

unpinned #6