Show HN: Hacker News archive (47M+ items, 11.6GB) as Parquet, updated every 5m (huggingface.co)

Article summary

The article presents a dataset of the complete Hacker News archive, containing every story, comment, and other item submitted to the site since 2006. The dataset is updated every 5 minutes and is available in Parquet format. It can be used for various purposes such as research, analysis, and training, and is intended to provide a comprehensive view of the technology community's discussions. The dataset is sourced from the ClickHouse Playground, which mirrors the official Hacker News Firebase API.

Main themes

• Hacker News dataset
• Data archiving
• Technology community
• Data analysis
• Parquet format
• Data licensing

What commenters say

• The dataset's license and terms of use are unclear and may not comply with regulations such as GDPR, which could lead to legal issues.
• Some commenters believe that if content is publicly available on the internet, it should be freely usable, while others argue that the original poster's rights and intentions should be respected.
• The dataset's design and structure have been criticized, particularly with regards to the representation of boolean values as integers, which may lead to inefficiencies in storage and analysis.
• There are differing opinions on how to handle deleted or moderated comments in the dataset, with some arguing that they should be preserved for completeness and others suggesting that they should be removed to maintain data quality.
• Some commenters have expressed concerns about the potential for the dataset to be used for commercial purposes without proper attribution or compensation to the original creators.
• The dataset's size and scope have led to discussions about the best ways to store and compress the data, with some suggesting that alternative formats or approaches may be more efficient.