Perplexity is using stealth, undeclared crawlers to evade no-crawl directives (blog.cloudflare.com)

Article summary

Perplexity, an AI-powered answer engine, has been observed using stealth, undeclared crawlers to evade website no-crawl directives, ignoring robots.txt files and rotating IPs to hide their crawling activity. This behavior is seen as a breach of trust and incompatible with the preferences of the internet community. Cloudflare has de-listed Perplexity as a verified bot and added heuristics to block this stealth crawling. The incident highlights the importance of transparency and respect for website directives in web crawling.

Main themes

• Web Crawling
• AI Ethics
• Robots.txt
• Website Security
• Internet Governance

What commenters say

• Perplexity's behavior is unacceptable and undermines the trust between websites and crawlers, as it ignores robots.txt files and uses stealth tactics to evade detection.
• The distinction between crawling and retrieving content on behalf of a user is crucial, and Perplexity's actions blur this line, making it difficult to determine their true intentions.
• Some argue that Perplexity's actions are a necessary step for AI development and that website owners should not have control over how their content is used, while others believe that this is a clear violation of website owners' rights.
• The use of cached data by AI companies like Perplexity raises concerns about the potential for scraping and the exploitation of website content without permission or compensation.
• There is a need for clearer guidelines and regulations on web crawling and AI training data, as the current situation creates uncertainty and conflict between website owners and AI companies.
• Some commenters argue that Perplexity's behavior is no different from other AI companies and that the issue is not unique to them, while others see this as a symptom of a larger problem with the AI industry's approach to data collection and use.