Cloudflare crawl endpoint (developers.cloudflare.com)

Article summary

Cloudflare has introduced a new API endpoint called /crawl, which allows developers to crawl entire websites with a single API call. The endpoint uses Browser Rendering to render pages in a headless browser and returns the content in multiple formats, including HTML, Markdown, and JSON. The endpoint respects robots.txt directives and AI Crawl Control, making it a well-behaved bot. It is available in open beta on both the Workers Free and Paid plans.

Main themes

• Cloudflare API
• Web Crawling
• Robots.txt
• AI Crawl Control
• Content Monitoring
• Internet Centralization
• Scraping Countermeasures
• Caching and Performance

What commenters say

• The new API endpoint could be useful for monitoring website content and synthetic monitoring.
• Some commenters think Cloudflare should offer a pre-scraped version of websites that use their proxy.
• Others argue that providing a pre-scraped version of websites could create a conflict of interest and centralize the internet.
• There are concerns that the new endpoint could bypass Cloudflare's own anti-AI crawl measures and give an unfair advantage to those who can afford it.
• Some commenters believe that the endpoint is a necessary addition to the crawling space and can help reduce the strain on developers' resources.
• Others think that Cloudflare is selling scraping countermeasures while also selling scraping services, which could be seen as hypocritical.
• There are also discussions about the potential impact on website owners who want to limit access to their content and the role of caching in reducing the load on origin servers.