Post Mortem on Cloudflare Control Plane and Analytics Outage (blog.cloudflare.com)

Article summary

Cloudflare experienced an outage of its control plane and analytics services due to a power failure at one of its data centers in Oregon. The incident lasted from November 2 to November 4, with most services restored after the first day. The company attributes the outage to a combination of factors, including a catastrophic failure of the data center's power systems and non-obvious dependencies in its own systems. Cloudflare is taking steps to improve its high availability systems and prevent similar outages in the future.

Main themes

• outage post-mortem
• high availability systems
• disaster recovery
• transparency and accountability
• reliability and availability
• cloud infrastructure resilience

What commenters say

• Cloudflare's transparency in its post-mortem analysis is commendable and sets a high standard for the industry.
• The company's decision to extend the outage to ensure rest and proper recovery was the right call, despite the impact on customers.
• Blaming the data center provider, Flexential, may not be entirely fair, as Cloudflare's own system design and lack of disaster recovery testing contributed to the outage.
• The incident highlights the tension between moving fast and ensuring reliability, and Cloudflare must find a balance between these competing priorities.
• Some commentators argue that Cloudflare should not have included speculation about the cause of the power failure in its post-mortem analysis, and instead should have waited for Flexential to provide its own account.
• Others believe that Cloudflare's transparency, including its speculation about the cause of the power failure, is necessary for learning and improvement.
• The outage has raised concerns about Cloudflare's overall reliability and availability, with some customers experiencing frequent outages and questioning the company's ability to meet its service level agreements.