A year of fighting scrapers on my 1.5 million-page website (patronview.com)

Article summary

The author of a 1.5 million-page website shares their experience fighting scrapers and bots, which account for 99% of their traffic. They tried various methods, including blocking certain countries and using Cloudflare's security features, and found that challenging suspicious traffic rather than blocking it was more effective. The author notes that the cost of scraping has decreased, making it a growing problem. They also mention that Cloudflare's pay-per-crawl feature could be a potential solution.

Main themes

• Scraping and bots
• Website security
• Cloudflare features
• Traffic management
• Economic solutions

What commenters say

• There is a difference between occasional scraping and constant, rapid re-scraping of the same site by bots.
• The use of Cloudflare can be effective in blocking bots, but some argue that it centralizes the internet and creates new problems.
• Hosting a website on a VPS without Cloudflare can be a viable option, but it may not be suitable for all use cases, especially those with high traffic or bandwidth requirements.
• Optimizing software and using simple bot tests can help reduce the impact of bots on a website, but some argue that this is not enough to completely solve the problem.
• The cost of bandwidth is no longer a significant issue for many websites, and unmetered bandwidth options are available, but others disagree, citing their own experiences with high bandwidth usage.
• Some argue that the bot problem is not as significant for personal websites, and that a $5/month VPS can be sufficient, while others claim that this is not enough to handle the traffic and bot activity.
• Using caching and optimizing database queries can help improve website performance and reduce the impact of bots, but this may not be enough to completely prevent issues.
• Serving a JavaScript page that sets a cookie and refreshes can be an effective way to stop bots, but this may not be a foolproof solution.