An update on residential proxies and the scraper situation (lwn.net)

Article summary

The article discusses the issue of residential proxies and scraper attacks on websites, where attackers use compromised devices to scrape website content. These proxies are often used by companies that offer 'ethically sourced' IP addresses, but in reality, they are used to attack websites. The article also mentions that some companies, like Bright Data, offer a 'free' VPN service that turns users' devices into residential proxies. The problem is difficult to solve, and website operators are scrambling to defend their sites without impeding legitimate traffic.

Main themes

• Residential proxies
• Scraper attacks
• Website defense
• Ethics of data collection
• AI and machine learning
• Internet security

What commenters say

• The use of residential proxies for scraping is a significant problem that needs to be addressed through systemic changes, rather than relying on individual users to take action.
• Some commenters argue that the issue is not with scraping itself, but with the lack of informed consent from device owners whose devices are being used as proxies.
• Others believe that the solution lies in creating a better, more accessible common crawl, which would reduce the incentive for scrapers to target individual websites.
• There is disagreement over whether banning or restricting access to certain types of content is an effective or desirable solution to the problem of scraper attacks.
• Some argue that the real issue is the companies that are driving the demand for scraped data, and that they should be held to a higher ethical standard.
• Others suggest that website operators could make their content more easily accessible, such as through a static zip file, to reduce the incentive for scraping.
• There is also a discussion about the effectiveness of current methods for defending against scraper attacks, and the need for more innovative solutions.
• Some commenters express concern that efforts to restrict scraping could have unintended consequences, such as injuring the open web or putting too much power in the hands of companies like Cloudflare.