TSV Utilities: Command line tools for large, tabular data files (github.com)

Article summary

eBay's TSV Utilities are a set of command line tools for manipulating large tabular data files, providing features such as filtering, statistics, sampling, and joins. The tools are designed to work with large files and are faster than other similar tools. They follow common Unix conventions and are intended to complement traditional Unix command line utilities. The tools are written in D and are available on GitHub.

Main themes

• command line tools
• data manipulation
• TSV files
• data analysis
• Unix utilities

What commenters say

• Some commenters appreciate the availability of command line tools for data manipulation and recommend other similar tools, while others prefer to use programming languages like Python. The use of delimiter-separated value files is seen as problematic by some due to their underspecification and potential for incorrect implementation. Some commenters argue that using safer and more well-specified file formats like AVRO would be preferable, but others point out that CSV is a de facto standard for exchange with spreadsheet programs. The choice of tool for data manipulation depends on personal preference and the specific task at hand, with some commenters preferring tools with a SQL interface and others preferring more traditional Unix utilities. Some commenters note that while new tools may offer useful features, they often can't replace the simplicity and ubiquity of established tools like awk.