What happened to TheNumbers.com (stephenfollows.com)

Article summary

TheNumbers.com, a website providing film data, suddenly went offline and later returned with reduced functionality due to being overwhelmed by AI bots scraping its data. The site's founder attributes the issue to the rise of agentic AI and the increasing difficulty of managing traffic from bots. The incident highlights the fragility of websites in the face of large-scale automated traffic. The site is now being rebuilt with a focus on supporting multiple types of users, including humans, search engines, and AI bots.

Main themes

• AI scraping and website fragility
• agentic AI and traffic management
• prediction markets and front-running
• web architecture and data sharing
• regulation and fairness
• website ownership and user expectations
• technological solutions and workarounds

What commenters say

• Some commenters believe that AI scraping is not a significant problem and that website owners should adapt to the new reality.
• Others argue that the issue is not just about scraping, but also about potential vulnerabilities that can be exploited for malicious purposes.
• There is a debate about whether bots should be allowed to make purchasing decisions, with some arguing that it could be a viable solution for websites to stay afloat.
• Some commenters suggest that the problem can be solved by using static site generators, caching, and content delivery networks.
• Others propose that prediction markets should implement measures to prevent front-running, such as locking down trading before a market closes.
• A few commenters argue that the issue is not just about technology, but also about the perceived fairness of prediction markets and the need for regulation.
• Some believe that website owners have the right to manage their own sites and data, and that users should not expect them to provide unlimited access to their content.
• Others think that the rise of AI-powered scraping is a symptom of a larger issue with the web's architecture and the need for a fundamental rethink of how data is shared and accessed.