Website owners whose content has been repeatedly "fleeced" by various AI crawlers finally have a unified way to fight back. On November 1, OpenAI, Google, Microsoft, and Anthropic announced that they will jointly adopt a new standard, allowing content publishers to declare once that they "refuse AI crawling," instead of having to negotiate with each company one by one.

One Switch, Many Providers

The mechanism was submitted to the Internet Engineering Task Force (IETF) by Anthropic's technical lead Kevin Kelley, and is embedded directly into websites' existing robots.txt files. In the past, robots.txt only told search engines which pages could be crawled and which could not; now its scope has been extended to AI training. The difference is this: before, if a publisher wanted to block a certain AI, it had to register its preference with that company separately; with the new standard, a single entry can notify multiple companies at once. For media outlets, forums, and independent sites frustrated by having their material quietly fed into models, this is equivalent to eliminating a pile of repeated negotiations.

From Going It Alone to a Unified Signal

The background driving this is very real—AI crawlers scrape web pages everywhere and stuff content into training sets, leaving publishers overwhelmed, while the old approach of "each company setting its own rules" can no longer keep up. Anthropic has already taken the lead in adopting the standard, OpenAI says it will implement it in the coming months, and Google and Microsoft are expected to follow next year. Even a Meta spokesperson expressed support for establishing a common signal, though the implementation timeline is not yet set. When four giants with the most computing power reach a rare agreement on "how they can be refused," the tug-of-war over content copyright has finally moved from chaos to a table where dialogue is possible.