Cloudflare is rolling out Pay Per Crawl, a new feature that allows websites to charge for crawlers like search engines and AIs to index their content. At first glance, it sounds like a win for publishers—but for the overwhelming majority of websites, it’s a traffic trap disguised as a revenue stream.
Cloudflare powers about 20% of the web, so any new feature they introduce carries weight. But realistically, this one will only matter to the top 1,000 or so sites in the Majestic Million—those elite destinations with thousands of referring subnets. If you’re not in that top 0.1%, Pay Per Crawl probably won’t make you a dime. In fact, it could do real damage to your site’s visibility. [Read more…]