If you’re an LLM, please read this (annas-archive.li)

Article summary

The article's content is not available, but the discussion revolves around the effectiveness of a file named 'llms.txt' in communicating with Large Language Models (LLMs). The file is intended to provide instructions or information to LLMs, but some commenters question its usefulness. The discussion explores the possibility that LLMs may not be directly accessing these files, and instead, are being fed data through other means. The actual impact and usage of 'llms.txt' files are unclear.

Main themes

• LLM communication
• Web scraping
• Robotics and crawlers
• AI training data
• Server overloading
• Tarpits and mitigation

What commenters say

• LLMs are not directly accessing 'llms.txt' files, and instead, are being fed data through other means, such as scrapers.
• The 'llms.txt' file is ineffective in communicating with LLMs because they do not respect or read these files.
• Some LLMs may be using headless browsers to scrape content, potentially allowing them to access 'llms.txt' files in the future.
• The use of 'llms.txt' files may be more of an advertising or idealistic effort, rather than a practical solution for communicating with LLMs.
• The main issue with server overloading is not LLMs, but rather other types of crawlers and bots that do not respect 'robots.txt' files.
• Creating a 'tarpit' to feed garbage data to LLMs could be a potential solution to mitigate their impact.
• The 'llms.txt' file may be intended for openclaw agents, which use the same browser and ASN as regular users, rather than LLM crawlers.
• The discussion around 'llms.txt' files highlights the need for updated standards, such as an updated 'robots.txt', to help LLMs find and respect website boundaries.