Why Twitter didn’t go down: From a real Twitter SRE (matthewtejo.substack.com)

Article summary

A former Twitter Site Reliability Engineer (SRE) explains how Twitter's caching system, which handles most of the site's traffic, has remained operational despite significant staff reductions. The engineer attributes this to the team's prior automation and reliability work, including the use of tools like Aurora and Mesos. The caching system is designed to distribute traffic across multiple servers and data centers, allowing it to continue functioning even if some servers or entire data centers fail. This design, combined with careful capacity planning, has helped Twitter maintain its uptime.

Main themes

• Twitter's caching system
• software maintenance and entropy
• complex system failures
• Mastodon vs Twitter resilience
• planned obsolescence
• disaster recovery planning
• staff reductions and operational risk

What commenters say

• Some commenters argue that software, like physical systems, is subject to entropy and will eventually fail without maintenance, while others disagree, citing the stability of well-designed software.
• The discussion touches on the challenges of maintaining complex systems, with some arguing that even well-designed software can fail due to unforeseen interactions between components.
• There are differing opinions on the resilience of Mastodon instances compared to Twitter, with some arguing that Mastodon's decentralized nature makes it more vulnerable to failures, while others see it as more robust.
• Some commenters believe that the key to Twitter's continued operation is not just its technical design, but also the expertise of its remaining engineers, who will be crucial in addressing any issues that arise.
• The conversation also raises questions about the role of planned obsolescence in the software industry, with some arguing that it drives unnecessary changes and others seeing it as a natural consequence of evolving technology.
• The importance of having a plan for disaster recovery and maintenance is emphasized, with some commenters noting that Twitter's ability to recover from failures is due in part to its prior planning and investment in reliability.
• There are concerns about the potential consequences of Twitter's staff reductions, including the risk of increased downtime and decreased ability to respond to issues.
• Some commenters speculate about the potential for Twitter to cut features or migrate to managed services in order to reduce its operational load and maintain stability.