Nonprofit organization Creative Commons (CC) has recently expressed cautious support for the "Pay-for-Crawl" technology, a system that automatically pays compensation when machines such as artificial intelligence web crawlers access website content.

Earlier this year, CC announced the framework of an "Open AI Ecosystem," aiming to provide legal and technical support for sharing datasets between companies controlling data and AI providers using the data for training. CC is best known for its contributions to licensing agreements, which allow creators to share their works while retaining copyright.

Hacker Leak

 The Necessity of Pay-for-Crawl

The concept of "Pay-for-Crawl," represented by companies such as Cloudflare, requires AI robots to pay websites each time they scrape content for model training and updates.

CC stated in its blog post: "If implemented responsibly, pay-for-crawl can become a way for websites to maintain their content creation and sharing... allowing content to be publicly accessible, otherwise it might not be shared or would be behind stricter paywalls."

The shift comes amid the "devastating impact" of AI on traditional publishers. In the past, websites allowed search engines like Google to crawl content in exchange for search traffic and clicks. However, with the popularity of AI chatbots, consumers now get answers directly from AI, leading to a sharp decline in website search traffic, seriously affecting publishers' profits.

The "pay-for-crawl" system offers small online publishers a way to recover from the AI impact, especially for those who lack the strength to negotiate one-time content agreements with major players like OpenAI, Google, and Meta. Currently, OpenAI has reached several significant partnerships with Conde Nast Group, Axel Springer Group, Perplexity, and Gannett Group, among others.

 CC's Reservations and Responsible Principles

Although expressing support, CC also raised some reservations, pointing out that such systems could lead to excessive concentration of power on the web and may hinder access to content for researchers, non-profits, cultural institutions, educators, and other organizations serving the public interest.

Therefore, CC proposed a set of responsible "pay-for-crawl" principles, including:

  • Pay-for-crawl should not be set as the default setting for all websites.

  • It should avoid one-size-fits-all rules for the entire web.

  • The system should allow throttling rather than just blocking, and ensure public access.

  • The system should be open, interoperable, and built using standardized components.

Industry Standards and Participants

In addition to Cloudflare, Microsoft is also building an AI marketplace for publishers, while smaller startups such as ProRata.ai and TollBit have begun entering this field.

Another organization, RSL Collective, has released a new standard called "Really Simple Licensing (RSL)