AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage (twitter.com)

Article summary

AWS has switched from gzip to zstd compression for S3 storage, resulting in a 30% reduction in compressed storage. The exact details of the implementation are unclear, but it appears to be a change to the internal storage compression. This change is expected to have significant cost savings for AWS. The switch is likely due to the better compression ratio and faster compression speed of zstd compared to gzip.

Main themes

• compression algorithms
• cloud storage
• data compression
• cost savings
• CPU overhead
• storage planning
• hardware compression

What commenters say

• The 30% reduction in compressed storage refers to a reduction in data size, not the number of objects stored.
• Some commenters believe that AWS may not be able to compress customer data that is already encrypted or compressed.
• Others argue that AWS likely compresses and encrypts all data at rest, regardless of whether the customer has already done so.
• There is debate over whether the benefits of compression outweigh the added CPU overhead and potential impact on storage planning.
• Some commenters suggest that hardware compression could be used to mitigate the CPU overhead.
• Others argue that the latency and IO benefits of compression make it a worthwhile tradeoff.
• There is also discussion over whether the cost savings from compression would be significant enough to justify the change.
• The switch to zstd may also have implications for data integrity and compression of already-compressed files.