We saved millions in SSD costs by upgrading our filesystem (heap.io)

Article summary

Heap, a product analytics company, experienced performance issues with their Postgres database due to rapid data growth, which led to high disk utilization and performance degradation. They upgraded their filesystem from ZFS with lz4 compression to ZFS 2.x with Zstandard compression, which reduced storage usage by 21% and halved write operation duration. The change had no observable impact on query performance. The upgrade was motivated by the need to decrease disk utilization and avoid adding more nodes to their cluster.

Main themes

• filesystem optimization
• database performance
• storage compression
• SSD wear leveling
• copy-on-write semantics
• Postgres configuration

What commenters say

• Upgrading to ZFS 2.x with Zstandard compression was a cost-effective solution for reducing storage usage and improving write performance.
• The use of copy-on-write semantics at multiple layers, including Postgres, ZFS, and SSDs, can lead to increased write amplification and reduced performance.
• The effectiveness of the upgrade depends on various factors, including the specific use case, data growth rate, and system configuration.
• Some commenters questioned the claim that ZFS's copy-on-write semantics are worse for SSDs, given that SSDs themselves use copy-on-write for wear leveling.
• The choice of compression algorithm and filesystem can have a significant impact on storage usage and performance, and careful evaluation and testing are necessary to determine the best approach.
• The use of Zstandard compression reduced storage usage, but its impact on query performance was not significant.
• The upgrade was a temporary workaround to avoid increasing storage capacity, but it may not be a long-term solution as data continues to grow.