Cloudflare Launches Pay Per Crawl for AI Bots
Cloudflare opens a private beta charging AI crawlers $0.01 or more per page using HTTP 402 and Ed25519-signed request headers, with new sites blocking AI training by default on September 15.
6 min read
Cloudflare opened a private beta for Pay Per Crawl today, wiring a price tag directly into HTTP infrastructure. AI crawlers accessing participating publisher sites get a HTTP 402 Payment Required response unless they authenticate with a signed request and pre-declare willingness to pay. The minimum rate is $0.01 per successful content retrieval.
TL;DR
| Spec | Value |
|---|---|
| Status | Private beta, July 1, 2026 |
| Minimum price | $0.01 USD per successful crawl |
| Auth method | Ed25519 Web Bot Auth + HTTP Message Signatures |
| Publisher options | Allow / Charge / Block per crawler |
| Default change | Sept 15: new free sites block AI training by default |
| Beta access | cloudflare.com/paypercrawl-signup |
Why This Exists
Bots overtook humans on the web last month. On June 3, 2026, Cloudflare CEO Matthew Prince shared Radar data showing automated requests generate 57.5% of HTML traffic - the first time machines have held the majority. Of that verified bot traffic, 51.8% goes to AI model training. Only 2.6% of AI crawler requests put an actual human on a page.
The Crawl-to-Referral Gap
The gap between what AI companies take and what they send back is severe. Cloudflare's data shows Anthropic crawls 11,122 pages for every single referral it sends to publishers. OpenAI's ratio is 1,700 to one. Google, by comparison, crawls five pages per referral.
That gap is what Cloudflare calls a broken value exchange. Traditional search drove enough traffic to justify indexing. AI systems extract summaries and facts, and users never leave the interface. UK publishers have pushed back against this pattern since last year, but the tools to act on that pushback haven't existed at scale.
How Cloudflare routes crawler requests through its pay-per-crawl pricing layer.
Source: blog.cloudflare.com
How Pay Per Crawl Actually Works
The HTTP 402 Handshake
The system works through existing web infrastructure. When an AI crawler hits a Pay Per Crawl-enabled domain without payment headers, Cloudflare returns HTTP 402 with pricing details. The crawler can then retry with signed payment credentials. There's also a proactive flow where crawlers include a maximum price upfront: if the site price falls at or below that ceiling, content comes back on the first request.
A paid crawl looks like this:
GET /article/how-llms-actually-work HTTP/1.1
Host: publisher.com
signature-agent: <ed25519-public-key-jwk>
signature-input: sig1=("@method" "@path" "@authority"); keyid="<key-id>"
signature: sig1=:base64url-encoded-signature:
crawler-max-price: 0.05
crawler-exact-price: 0.02
HTTP/1.1 200 OK
crawler-charged: 0.02
Content-Type: text/html
Cloudflare acts as the Merchant of Record, handling billing aggregation and paying publishers monthly.
Web Bot Auth
Crawlers that want to participate must implement Web Bot Auth - an Ed25519 key pair where the public key is published as a JWK, and each request is signed using HTTP Message Signatures. The signature-agent, signature-input, and signature headers carry proof of identity and payment intent. The crawler-max-price and crawler-exact-price headers negotiate the rate.
This isn't something you can bolt on in an afternoon. Crawlers need to generate key pairs, publish their public keys at a well-known endpoint, update HTTP clients to sign every request, and integrate with Cloudflare's billing API. The barrier is intentional: it filters out commodity scrapers and forces serious AI companies to formally commit.
Publisher Setup
Configuration lives in the Cloudflare dashboard under AI Crawl Control > Settings > Pay Per Crawl. Enable the feature, set a price (floor: $0.01), and optionally create custom rules for different content paths. Publishers can also set per-crawler policies independently of pricing: Allow (free access), Charge (pay to crawl), or Block (no access at any price). A publisher can allow Googlebot for free, charge OpenAI's crawler, and block Anthropic completely.
Who's In
Ceramic.ai and You.com are the first publicly named AI partners in the Pay Per Use variant of this system. OpenAI, Anthropic, Google DeepMind, and Meta haven't announced Pay Per Crawl support. Without Web Bot Auth adoption from the major AI labs - which collectively create most of that 57.5% bot traffic - publishers who enable the feature will mostly see their content blocked outright rather than monetized.
The Human Native acquisition adds the other half of the picture. Human Native built tooling to convert unstructured web content into clean structured datasets for AI licensing. Folding that into Cloudflare gives the company both the publisher distribution and a data-processing pipeline to eventually run a proper content marketplace. Cloudflare has been building out its AI compute and agent infrastructure for months; Pay Per Crawl fits that direction.
Cloudflare's "Content Independence Day" announcement establishes new default policies for AI training access.
Source: blog.cloudflare.com
Where It Falls Short
Adoption chicken-and-egg. Pay Per Crawl only works if AI companies implement Web Bot Auth. Right now, two small companies have. The big labs haven't committed, and they can simply route around the system by not signing requests - which currently means getting blocked rather than paying.
Pricing floors. At $0.01 per crawl, a publisher with 10,000 crawler requests per day earns $100. For a news site paying six-figure hosting bills, that math doesn't work. Premium content - legal databases, medical research, financial data - needs dynamic pricing. The advanced configuration promises per-path pricing rules, but detailed documentation isn't published yet.
Private beta limits access. Signup requires a form submission or an Enterprise account executive. Most independent publishers - the long tail that actually needs this most - aren't in yet.
Opt-in inertia. The September 15 default change is the most concrete outcome from today's announcement. Starting then, new Cloudflare sites automatically block AI training on ad-supported content while allowing search indexing. But that only applies to new sites; existing customers must change settings manually. The fraction of the existing Cloudflare customer base that actively adjusts these settings will be small.
Cloudflare is the first infrastructure company with enough web distribution to act on the publisher-AI traffic imbalance at the protocol level rather than just publishing a statement. The September 15 default is real and automatic. Everything else depends on whether the major AI labs integrate Web Bot Auth rather than quietly treating cloudflare.com/paypercrawl as another scraping friction to work around.
Sources:
- Introducing pay per crawl: Enabling content owners to charge AI crawlers for access - Cloudflare Blog
- Content Independence Day: no AI crawl without compensation - Cloudflare Blog
- Cloudflare Moves To Make AI Pay For The Content It Consumes - Forbes
- What is Pay Per Crawl? - Cloudflare Docs
- The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals - Cloudflare Blog
- Human Native is joining Cloudflare - Cloudflare Blog
- Cloudflare's Human Native acquisition signals a new content economy for publishers - Digiday
- AI Crawler Bot Traffic Statistics 2026 - Digital Applied
