Cloudflare announced a new setting called "Disallow AI Training," which allows administrators to continue accepting search engine crawlers while rejecting website content for AI training. Apple, Google, and Microsoft have pledged to comply with this setting.

After website administrators enable this setting, Cloudflare will still allow mixed crawlers for search and AI training, provided the system marks them as "Accountable" (accountable). Other AI training crawlers will still be blocked. The official stated that blocking such crawlers could affect a website's search rankings.

Cloudflare said most customers do not need to make any changes: websites that previously selected "Block" or "Block on pages with ads" in Training will automatically migrate to "Disallow AI Training."

In addition, Google will launch the Google-Extended URL-level transparency tool in the coming weeks, Apple is also developing its own URL-level tool, and Microsoft plans to support rejecting AI training through robots.txt in early 2027.