Creepy Crawlies (people.kernel.org)

Article summary

The article discusses the issue of AI crawlers overwhelming the Linux kernel's git repository with requests, causing a significant load on the system. The crawlers are rendering commits as HTML, which is an inefficient way of accessing the data, instead of cloning the repository. The author has implemented a challenge-response system, Anubis, to slow down the crawlers, but they have adapted and are now solving the challenges. As a result, the repository is turning off features to reduce the number of crawlable URLs.

Main themes

• AI crawlers
• git repository overload
• inefficient data access
• challenge-response systems
• bot mitigation
• LLM training
• internet scraping
• data usage ethics

What commenters say

• Some commenters suggest that the crawlers should be using more efficient methods to access the data, such as cloning the repository.
• Others argue that the crawlers are not smart and are only focused on avoiding blocking.
• It is proposed that the difficulty of the Anubis challenge should be dynamic and depend on the IP's request rate or the age of the commit.
• Some commenters believe that the crawlers are not just used for training AI models, but also for other purposes such as tool-use LLMs.
• There is a suggestion that the repository should use a global rate limit to mitigate the bot traffic.
• Others think that the crawlers are a sign of a larger problem with the internet and the way data is being scraped and used.
• Some commenters are taking drastic measures, such as taking down their own cgit repositories, due to the overwhelming load caused by the crawlers.
• There is a concern that organized crime may also be involved in training LLMs and using crawlers to gather data.