Details of the Cloudflare outage on July 2, 2019 (blog.cloudflare.com)

Article summary

Cloudflare experienced a 27-minute outage on July 2, 2019, due to a poorly written regular expression in a WAF rule that caused CPU exhaustion. The company has written a detailed post-mortem analysis of the incident, explaining what went wrong and how they plan to prevent similar outages in the future. The analysis highlights the importance of transparency and the need for robust testing and deployment procedures. Cloudflare is taking steps to improve its WAF rule deployment process and prevent similar incidents.

Main themes

• Cloudflare outage
• WAF rule deployment
• Regular expression backtracking
• Transparency and accountability
• Testing and deployment procedures

What commenters say

• The outage could have been prevented with better deployment procedures, such as progressive deployments and canary releases.
• Cloudflare's use of Lua scripts and regular expressions in their WAF is a bad idea due to the potential for recursion attacks and backtracking.
• The company's decision to switch to a new regex engine, such as RE2, is a good move to prevent similar outages in the future.
• The incident highlights the importance of separating monitoring and infrastructure to prevent circular dependencies and ensure timely incident response.
• The use of a third-party platform for monitoring services is necessary to provide context and insight into outages.
• Regular drills and exercises can help prevent bit-rot and ensure that emergency procedures are effective.
• The development of Cloudflare's Lua-based WAF was likely influenced by compatibility with existing mod_security rules.
• Switching to a new regex engine is not a significant risk and can be tested and deployed within a relatively short timeframe.