Google Cloud Went Down Because It Was Misconfigured

A recent major outage of Google Cloud was caused because of a misconfiguration, not only because of network congestion, like Google initially reported. The outage was described by Google as “network congestion issue in eastern USA, affecting Google Cloud, G Suite, and YouTube”. It also caused services such as Shopify, Snapchat, and Discord to go down. Some people reported that they could not control the temperature in their home or apartment through Google Nest.

In the post-mortem incident report, Google specifically states that “Two normally-benign misconfigurations, and a specific software bug, combined to initiate the outage: firstly, network control plane jobs and their supporting infrastructure in the impacted regions were configured to be stopped in the face of a maintenance event.” Also, “the software initiating maintenance events had a specific bug.” So, as I understand it, maintenance was being done, and the misconfigurations, as well as a software bug, lead to this major outage.

Eligible For Credit

Because of the outage, you may be entitled to receive SLA credits. Google’s statement about the incident says that “If you believe your paid application experienced an SLA violation as a result of this incident, please populate the SLA credit request: https://support.google.com/cloud/contact/cloud_platform_sla

A full list of all Google Cloud Platform Service Level Agreements can be found at https://cloud.google.com/terms/sla/.

For G Suite, please request an SLA credit through one of the Support channels: https://support.google.com/a/answer/104721

G Suite Service Level Agreement can be found at https://gsuite.google.com/intl/en/terms/sla.html”

This Google Cloud outage was very significant, as it had an effect on a lot of services, from Nest to Gmail to YouTube and Vimeo, all services that rely on Google’s network. I realize that misconfigurations and software bugs do happen, despite our best efforts to do things that we think will prevent such occurrences. Just goes to show you that backups are necessary, even if you have to download them and put them on a hard drive in your office. If your website were to go down, you should have a backup not only on the Google Cloud, but perhaps in several places, as well as on a hard drive. Because, well, you never know.

Image credit: Gizmodo and DownDetector.