CDC File Transfer (github.com)

Article summary

The CDC File Transfer project provides tools for syncing and streaming files from Windows to Linux, utilizing Content Defined Chunking (CDC) to optimize data transfer. The tools, cdc_rsync and cdc_stream, aim to improve upon traditional file transfer methods like rsync. The project originated from Google's Stadia and is now open-sourced. The tools are designed to work efficiently with large files and multiple small files, reducing transfer times.

Main themes

• CDC File Transfer
• Content Defined Chunking
• rsync comparison
• file transfer optimization
• open-sourcing
• legal implications
• bittorrent usage
• large-scale storage systems
• Google Stadia project

What commenters say

• Some commenters suggest replacing rsync with the new tool, while others argue that rsync's fixed-size chunks have advantages.
• The CDC algorithm is not new, but the implementation and open-sourcing of it are valuable contributions.
• There are potential legal implications of using chunk-based deduplication in large-scale storage systems, particularly regarding chain of custody and piracy.
• The use of bittorrent for legitimate purposes, such as distributing research data, is highlighted, but its association with piracy is also acknowledged.
• Some argue that replacing rsync with the new tool may not be necessary, as rsync's algorithm is more complex and optimized than initially described.
• The project's origin from Google's Stadia and its open-sourcing are seen as a positive example of the company's track record of open-sourcing shutdown projects.
• The potential for applying CDC to large-scale blob storage, such as Amazon S3 or Google Drive, is discussed, with some noting the need for metadata storage and chunk deduplication.