A routine gem update ended up creating $73k worth of subscriptions (serpapi.com)

Article summary

A routine gem update led to a major issue where 474 new subscriptions were created, resulting in $73,271.36 in unexpected charges. The problem was caused by a change in the behavior of the Mongoid library, which was not properly accounted for in the application's code. The company rolled back the update, refunded the affected customers, and apologized for the mistake. The incident highlights the importance of careful testing and consideration of dependency updates.

Main themes

• dependency updates
• breaking changes
• testing and validation
• NoSQL databases
• data integrity
• semantic versioning
• responsibility and accountability

What commenters say

• The maintainers of the Mongoid library are to blame for introducing a breaking change without properly reflecting it in the versioning.
• The company should have tested the update more thoroughly before deploying it.
• The change in Mongoid's behavior was a necessary step to align with ActiveRecord, but it was poorly handled.
• The incident demonstrates the risks of relying on third-party dependencies and the importance of careful testing and validation.
• The company's response to the incident, including rolling back the update and refunding customers, was appropriate and responsible.
• The use of NoSQL databases like MongoDB can lead to problems with data consistency and integrity.
• Dependency updates should be carefully evaluated and tested before deployment, even if they seem minor.
• The semantic versioning system is not always effective in preventing breaking changes from being introduced in minor version updates.