Why wordfreq will not be updated (github.com)

Article summary

The author of the wordfreq project, which provides word frequency data, has announced that the project will not be updated due to the increasing presence of generative AI-generated text on the internet, which skews the data. The author also cites the loss of access to free data sources, such as Twitter and Reddit, which have either shut down or begun selling their data. As a result, the author has decided to freeze the current version of wordfreq, allowing it to become outdated over time rather than risking the introduction of inaccurate data. The author expresses frustration with the dominance of generative AI in the field of natural language processing.

Main themes

• Generative AI impact on data
• Loss of free data sources
• Natural language processing
• Data pollution
• Open-source projects
• AI dominance in NLP

What commenters say

• The increasing presence of generative AI-generated text on the internet makes it difficult to collect reliable word frequency data that reflects human language use.
• The loss of access to free data sources, such as Twitter and Reddit, has made it challenging to update the wordfreq project.
• Some commenters suggest that collecting data with the caveat that it doesn't represent human output could be a viable alternative, while others argue that this approach would still be problematic.
• There is a concern that the dominance of generative AI in NLP is stifling other approaches and making it difficult for researchers to work on projects that are not related to AI.
• The concept of 'low-background steel' is used as an analogy to describe the need for 'unpolluted' data, free from AI-generated content.
• Some commenters disagree on the feasibility of creating a new dataset that is free from AI-generated content, with some arguing that it is possible and others claiming that it is not.
• The discussion touches on the topic of biodiversity and the importance of preserving human-generated content in the face of increasing AI dominance.
• There is a sense of frustration and pessimism among some commenters regarding the impact of AI on the field of NLP and the potential consequences for human-generated content.