Amazon's AI crawler is making my Git server unstable (xeiaso.net)

Article summary

A Git server is experiencing instability due to Amazon's AI crawler, which is aggressively scraping the website despite measures to prevent it. The administrator has set up a proof-of-work scheme to protect the server, but it's a temporary solution. The issue is not unique to this case, as many websites are experiencing similar problems with AI and SEO bots. These bots are barely respecting robots.txt directives and are causing high server loads.

Main themes

• AI crawler issues
• Server instability
• robots.txt respect
• Web scraping
• Proof-of-work schemes
• Bot blocking

What commenters say

• The current state of AI crawlers is unsustainable and will lead to universal blocks if they do not change their behavior and respect website directives.
• Implementing a global tarpit or similar solution could be an effective way to mitigate the issue, but it may not be feasible for all websites.
• Some argue that intellectual property law should be sufficient to prevent large-scale data theft, but it is not being effectively enforced against AI companies.
• Others propose that websites could sell their data to AI bots in bulk, rather than being scraped, but this is unlikely to happen as AI companies prefer to get data for free.
• The use of proof-of-work schemes, such as Anubis, can help protect servers from aggressive scraping, but may not be a long-term solution.
• Some commenters suggest that feeding AI bots nonsense data or using honeypots could be a way to deter them, but this may not be a scalable or effective solution.
• The issue is driving some websites to move away from the open web and into closed ecosystems, which could have negative consequences for the internet as a whole.
• There is a need for a more comprehensive solution, such as a single web index, to avoid crawling-induced DDoS attacks and ensure the sustainability of the web.