Google Cloud networking issues in us-east1 (status.cloud.google.com)

Article summary

Google Cloud experienced networking issues in the us-east1 region due to physical damage to fiber bundles, resulting in elevated latency and errors. The issue was mitigated through elective routing and eventually resolved after repairs were made. The incident lasted from July 2 to July 3, 2019. Google Cloud has promised to perform an internal review and make improvements to prevent similar incidents in the future.

Main themes

• Google Cloud outage
• networking issues
• fiber damage
• cloud reliability
• multi-region deployment

What commenters say

• Running a service in multiple regions can help mitigate the impact of outages, but may not be sufficient to prevent all issues.
• Using multiple cloud providers can provide additional redundancy, but may be impractical due to high egress costs.
• Google's internal systems and services may not be hosted on Google Cloud Platform, which could lead to a lack of dogfooding and testing.
• The cause of the outage was attributed to physical damage to fiber bundles, highlighting the potential for external factors to impact cloud services.
• Some argue that using multiple cloud providers is not a viable strategy for most businesses, and that investing in infrastructure-as-code is a more effective way to ensure high availability.
• Others believe that running a service in multiple regions or providers can help prevent downtime, but may require significant additional complexity and cost.
• The incident highlights the importance of diversity in infrastructure and systems to prevent cascading failures and ensure high availability.