OpenZFS deduplication is good now and you shouldn't use it (despairlabs.com)

Article summary

The article discusses the new 'Fast Dedup' feature in OpenZFS 2.3.0, which aims to improve the performance and efficiency of data deduplication. The author explains the challenges of traditional deduplication, including high memory requirements and performance overheads, and how Fast Dedup addresses these issues. The feature reduces the memory footprint of the live entry list and provides tools to manage the dedup table contents. Despite these improvements, the author advises against using deduplication due to its complexity and potential drawbacks.

Main themes

• ZFS deduplication
• Fast Dedup feature
• Performance optimization
• Memory management
• Data storage efficiency

What commenters say

• Inline deduplication can save a significant amount of write IO, but it requires the dedup table to fit in RAM to be effective.
• Offline or lazy deduplication would be a useful feature, allowing for deduplication without impacting write performance.
• Some users argue that deduplication is not necessary and that resources would be better spent on adding parity or backing up important files.
• Others suggest that deduplication can be useful for specific use cases, such as archiving large amounts of data with many duplicates.
• The Windows deduplication methodology, which uses a background process to deduplicate data, is seen as a possible alternative approach.
• Some commenters express concerns about the reliability and potential data corruption issues with deduplication, particularly in cases where data is highly duplicated.
• The ability to deduplicate data after the fact, as a cron job or offline process, is seen as a desirable feature by some users.