When AI chatbots provide answers directly, users no longer click through to original web pages—the "traffic collapse" sparked by generative AI is putting global content creators in a survival crisis. In this context, the non-profit organization Creative Commons (CC), long known for advocating open sharing, has made an unusual statement: cautiously supporting "pay-to-crawl" technology, aiming to rebuild a sustainable economic model for content ecosystems.

CC is well-known for promoting the Creative Commons licensing (CC licenses). In July of this year, it proposed building a legal and technical framework for AI data sharing. Now, facing the reality that AI crawlers are extensively scraping content without returning traffic, CC admitted in its latest blog post: "If implemented responsibly, pay-to-crawl could help websites maintain content creation and public sharing, preventing more content from being locked behind paywalls or disappearing altogether."

The core logic of "pay-to-crawl" is simple and straightforward: when AI companies (such as OpenAI, Google, and Meta) crawl websites to train their models, they must pay content providers. This model was pioneered by Cloudflare, reflecting the collective anxiety of the content industry—the traditional internet contract of "content for traffic" has collapsed. In the past, websites were happy to be indexed by search engines because search results brought traffic and advertising revenue. Today, however, after getting answers from AI conversations, users almost never click through to the original pages. According to multiple studies, search traffic for news publishing has plummeted, with small and medium-sized media outlets hit the hardest.

Larger media outlets can still negotiate exclusive licensing agreements with AI giants—such as OpenAI with Condé Nast and Axel Springer, Perplexity with Gannett, Amazon with The New York Times, and Meta with several publishers—but millions of independent blogs, small news sites, and educational platforms lack such leverage. If pay-to-crawl becomes standardized and automated, it could become a source of "inclusive income" for them.

However, CC's position is not entirely optimistic. The organization clearly warns that if poorly designed, pay-to-crawl could **exacerbate the concentration of online power** and even cut off access paths for public interest entities. "Researchers, non-profits, cultural heritage institutions, educators, and other groups serving the public interest should not be blocked by paywalls," CC emphasized.

To address this, CC has proposed a set of "responsible pay-to-crawl" principles:

- It should not be the default option on websites; it must be enabled by content providers themselves;

- Avoid one-size-fits-all global rules, respecting website autonomy;

- Support "throttling" rather than "blocking," allowing low-frequency or non-commercial crawling;

- Ensure access for public interest, such as academic and educational purposes;

- The system should be open-source, interoperable, and use standardized components.

Notably, the pay-to-crawl ecosystem is rapidly taking shape. In addition to Cloudflare, Microsoft is building an AI content market for publishers; startups like ProRata.ai and TollBit are also developing similar solutions. More importantly, a new standard called **RSL** (Really Simple Licensing) has gained widespread support. This standard allows websites to declare which content can be crawled and for what purposes, but it does not force crawlers to stop—offering a middle path of "declaration as authorization." The three major CDN giants Cloudflare, Akamai, and Fastly have adopted RSL, and content providers such as Yahoo, Ziff Davis, and O’Reilly Media have also joined. CC has also announced support for RSL and included it in its "CC Signals" project—a program aimed at developing new content licensing and discovery tools for the AI era.