Cloudflare is rolling out Pay Per Crawl, a new feature that allows websites to charge for crawlers like search engines and AIs to index their content. At first glance, it sounds like a win for publishers—but for the overwhelming majority of websites, it’s a traffic trap disguised as a revenue stream.
Cloudflare powers about 20% of the web, so any new feature they introduce carries weight. But realistically, this one will only matter to the top 1,000 or so sites in the Majestic Million—those elite destinations with thousands of referring subnets. If you’re not in that top 0.1%, Pay Per Crawl probably won’t make you a dime. In fact, it could do real damage to your site’s visibility.
Should Your Site Use Pay Per Crawl? A Quick Guide:
| Website Type | Recommendation |
|---|---|
| Top 1,000 sites in Majestic Million | Wait and See |
| Publisher (not top 1,000) | No – Loss of traffic likely |
| Small business site | No – Not worth the risk |
| Corporate site | No – Open access recommended |
| Company website | No – Crawling supports discoverability |
| Ecommerce site | No – You need product visibility |
| Hobby site or personal blog | No – You’ll be ignored by bots |
| Independent content creator | No – Traffic loss inevitable |
| Aggregator/AI-powered tool | Maybe – But tread carefully |
Pay Per Crawl might work for the Reddit-level players—the sites search engines must include because users expect them there. As one Googler once put it, “there are certain websites that must be in our search results.” Those are the kinds of sites that might have leverage to negotiate crawl fees. Most others? Not so much.
If you’re a blogger, run a niche hobby site, or even a mid-size content publisher, enabling this feature will likely result in your content disappearing from search and AI results. Why? Because they won’t pay. They’ll skip you. And with so many alternatives online, your absence won’t even be noticed.
Search engines and AIs care about serving good results—not about your individual site’s bottom line. They’ll find other sources. Meanwhile, you’ll be losing the free traffic that you probably depend on. That trickle of crawl fee income? It won’t come. Or if it does, it won’t offset the revenue and visibility you just lost.
Remember when Pay Per Inclusion sounded good? Exactly.
It’s a Novel Idea
It’s definitely a novel idea, and since 20 percent of websites use Cloudflare, it will have an impact. But that impact will ultimately only affect about 1 percent or less of those 20 percent, in my opinion. In other words, the top 1,000 or less of the Majestic Million websites. (Majestic Million lists the top websites on the internet, the ones with the most referring subnets).
Unless you’re one of those top 1,000 sites, don’t expect a payday or to “get rich” off of Pay Per Crawl.
The only websites that would possibly benefit from this are those websites that the search engines and AIs feel as if the site “must be there”. I once heard a Googler say, “there are certain websites that must be in our search results, as our users expect them to be there”. That’s very true. So, if those sites decide to use the “pay per crawl” model, then search engines or AIs “may” pay. They may also negotiate with you, as they did with sites like Reddit.
But if you’re an independent publisher, a blogger, or have a personal or hobby website, and you decide to try out Cloudflare’s Pay Per Crawl option, you’ll just lose traffic. The search engines and AIs just don’t care. They don’t care if your website is in their index or not. It doesn’t affect them, or their results. There are plenty of other websites that will take your place, others that have the content that search engines and AIs need in order to provide good results for their users. Let me say this again:
The search engines and AIs don’t care about your website. They won’t be paying your fee to crawl your website. Unless, of course, you’re one of the top websites on the internet or one of the top websites in your industry or niche. I recall Google reaching out to website owners of certain websites because those sites were blocking crawlers from crawling their websites in robots.txt. But as I mentioned, those were sites that ‘should be’ in Google’s index, and Google’s users expected to see them. If you block Google from crawling, and you’ve verified the site in Search Console, you may get an automated email from them—but they won’t be reaching out to you personally unless you’re a large, high traffic site. One of the top 1 percent on the web.
Inevitably, search engines like Google and Microsoft Bing, and AIs will take an official stance and position on Pay Per Crawl. They have to, as websites using Cloudflare are a significant part of the internet. But honestly, I think this will end up having the same fate as Pay Per Inclusion—and history shows how that went.
My Recommendations:
- If you run an ecommerce, company, or corporate site: Allow all crawlers. You want that visibility.
- If you’re a publisher outside the top 1,000: Still allow all crawlers. Don’t trade reach for a fee that may never arrive.
- If you’re in the top tier of the web: Wait and see. Give it 6–12 months. Watch how search engines and AIs respond.
- For everyone else: Just turn it OFF.
Let the big players experiment and absorb the risk. Don’t bet your traffic on a feature that’s more theoretical than practical.