llms.txt: The "robots.txt" for AI Agents

Imagine a "No Entry" sign for artificial intelligences. llms.txt is exactly that: a file websites use to tell AI agents where they can snoop and where they can't. It's a way to regain some control over who or what reads your content.

It works like the old robots.txt, but it's designed for new AI "agents" that don't just index pages. These agents, like those that summarize pages or answer complex questions, need more specific rules. The goal is to prevent misuse or unwanted use of data.

Until now, AI agents could roam almost undisturbed, collecting data left and right. This created privacy issues, server overload, and, let's face it, a bit of chaos. llms.txt offers a solution to rein in this wild freedom.

Websites can specify rules for different "User-Agents" (i.e., different AI bots), deciding whether to "Disallow" or "Allow" access to certain sections. They can also provide a contact, in case the AI misbehaves. The AIE World's Fair 2026 website, for instance, has already implemented an llms.txt file to guide AI agents.

It's a simple text file, placed in the root directory of your website, just like robots.txt. Inside, you write the instructions, much like a rulebook for your digital guests. You don't need to be a programmer to understand it; a few lines of commands are enough.

You can say "hey Perplexity, don't look here" or "ChatGPT, you can only read this section." It's a step forward for transparency and data management. It will help us better understand who accesses our online content and why, making the web a slightly more organized place.