Ban me at the IP level if you don't like me (boston.conman.org)

Article summary

The article discusses the issue of aggressive web scraping and the challenges of blocking malicious traffic. The author encounters a webbot called Thinkbot, which claims to be in a test phase and invites blocking at the IP level. The author discovers that Thinkbot is using multiple IP addresses from various network blocks owned by Tencent, leading to the conclusion that blocking individual IP addresses may not be effective. The author decides to block entire network blocks to mitigate the issue.

Main themes

• Web scraping
• IP blocking
• Malicious traffic
• Network security
• ISP responsibility
• Cloud providers

What commenters say

• Blocking individual IP addresses may not be effective against aggressive web scraping due to the use of multiple IP addresses and residential proxies.
• Whitelisting IP ranges or entire countries may be a more effective approach, but it can be challenging to maintain and update such lists.
• ISPs should be responsible for policing their users' traffic and responding to abuse complaints to prevent malicious activity.
• Blocking entire ISPs or countries may not be a viable solution due to the potential impact on legitimate users and the lack of competition in the broadband market.
• Using a combination of IP blocking and other factors, such as user behavior, may be a more effective way to prevent malicious traffic.
• The use of cloud providers' IP ranges can make it difficult to distinguish between legitimate and malicious traffic.
• Some commenters argue that ISPs should not be allowed to police traffic, while others believe it is necessary to prevent abuse.
• The issue of web scraping and malicious traffic is complex and requires a multi-faceted approach to address.