Lessons Learned from Twenty Years of Site Reliability Engineering (sre.google)

Article summary

Google's Site Reliability Engineering (SRE) team has learned several lessons over the past twenty years, including the importance of testing recovery mechanisms, having a 'big red button' to revert changes, and intentionally degrading performance modes to ensure a consistent user experience. The team has also learned to automate mitigations, reduce the time between rollouts, and maintain a diverse infrastructure to prevent single points of failure. These lessons were learned through various outages and incidents, including a 15-minute global outage of YouTube and a widespread packet loss in multiple datacenters. By sharing these lessons, the SRE team aims to help others improve their reliability and resilience.

Main themes

• Site Reliability Engineering
• Disaster Recovery
• Infrastructure Diversification
• Automation and Testing
• Communication Channels
• Outage Prevention
• Business Continuity Planning

What commenters say

• Having backup communication channels that are not dependent on the primary system is crucial in case of an outage.
• Testing and automation are essential for ensuring the reliability and resilience of complex systems.
• Relying on a single cloud provider or infrastructure can be a single point of failure, and diversification is necessary to prevent outages.
• The concept of a 'big red button' to revert changes is important, but it may not always be feasible or effective in all situations.
• The use of IRC servers as backup communication channels can be effective, but they may have limitations and require careful planning and testing.
• The idea of a 'meteor hitting a DC' is a metaphor for unexpected disasters, and having a plan for such events is necessary, but the likelihood and impact of such events should not be exaggerated.
• Diversification of infrastructure and services can help prevent single points of failure, but it also increases complexity and cost.
• The importance of having a plan for disaster recovery and business continuity cannot be overstated, and it requires careful consideration of various scenarios and risks.