TL;DR
On July 1, 2026, Cloudflare announced it is splitting AI bots into three categories — Search, Agent, and Training — and that from September 15, 2026 it will block Training and Agent bots by default on ad-displaying pages. Search crawlers stay allowed. Crawlers that blend search with training get blocked, because mixed use defeats the whole point of the split. The default applies to new customers, new zones, and all existing free customers — which is a very large slice of the web. Alongside it, Cloudflare is shipping Content-Usage signals in robots.txt (immediate, reference, full) and a "BotBase" dashboard for Enterprise.
Here's the part most coverage will get wrong: this is about bots, not about you. ScrapeMaster runs inside your own browser, in your own session, on a page you already loaded and can already see. It is not a datacenter crawler fetching pages you never visited. A default-block rule aimed at Training and Agent crawlers is not aimed at a person reading a page.
But we're not going to oversell that. Cloudflare's Agent category is aimed at browser-use agents — chat-fetch bots and AI models driving Chrome — and the line between "a person with a tool" and "an agent" is precisely the line the industry is currently arguing about. Anti-bot systems produce false positives. We don't rotate proxies or fingerprints and we can't solve CAPTCHAs. If a site blocks you, the honest answer is: it blocked you.
What Cloudflare actually announced
Strip the framing away and there are four moving parts.
1. Three bot categories instead of one
Until now, "AI bot" was one undifferentiated bucket, and blocking it was an all-or-nothing decision publishers hated making. Block AI bots and you might vanish from AI search. Allow them and you're donating your archive to a training run. Cloudflare's answer is to split the category by purpose:
| Category | What it does | Default from Sept 15, 2026 |
|---|---|---|
| Search | Crawls to index and link back to your page | Allowed |
| Agent | Fetches a page on behalf of a user's live request or task | Blocked on ad-displaying pages |
| Training | Collects content to train or fine-tune a model | Blocked on ad-displaying pages |
| Mixed-use | Blends search crawling with training collection | Blocked |
That last row is the sharp one. A crawler that says "I'm a search bot" but funnels the same fetches into a training corpus doesn't keep the search exemption. The exemption is for the behavior, not the user agent string. If you've ever wondered why bot-blocking felt useless — everyone just claimed to be a search bot — this is the fix.
2. The default flips, and it flips for free customers too
The important word is default. This isn't a feature you opt into; it's a rule that applies unless you turn it off. It covers new customers, new zones, and all existing free customers — not a rounding error, but a substantial portion of the web's small and mid-sized publishers. The "ad-displaying pages" scope narrows it somewhat, but the practical effect is that a large slice of the open web changes its posture toward AI crawlers on one specific date, without any individual publisher deciding anything.
That's the real story. Not the policy — the default. Defaults are where the internet's rules actually live.
3. Content-Usage signals in robots.txt
Cloudflare is shipping Content-Usage signals with three values:
immediate— you may use this content to answer a user's question right now.reference— you may cite and link to this content.full— broader usage, including training.
This deserves more attention than the blocking headline, because it's the first time "what may you do with this page" gets a vocabulary instead of a binary. robots.txt has only ever been able to say whether you may fetch — never what you may do with the thing you fetched. Those are different questions, and conflating them is why the AI-and-publishers fight has been so incoherent.
4. Pay Per Crawl became Pay Per Use
Cloudflare also reframed "Pay Per Crawl" as "Pay Per Use". The distinction matters more than it sounds:
- Pay Per Crawl = publishers get paid when a bot fetches the page.
- Pay Per Use = publishers get paid when the content shapes an AI answer.
Launch partners named are Ceramic.ai and You.com. And the justification for the change is the most interesting statistic in the whole announcement: Cloudflare says more than 50% of AI crawl traffic is re-fetches of pages that haven't changed.
Sit with that. Over half of the AI crawling hammering the web is bots re-downloading identical bytes. Paying per fetch would have meant paying for that waste — worse, incentivizing it. Paying per use severs payment from fetch entirely, which is fairer to publishers and quietly a huge argument for caching discipline.
Why this is about bots and not about you
Now the honest part, which is the reason this post exists.
There is a category difference between these two things, and it is not a marketing distinction:
| Datacenter crawler | In-browser extraction | |
|---|---|---|
| Who fetches the page | A server you don't control, in a datacenter | Your browser, on your machine |
| Whose session | An anonymous or synthetic session | Your own, already-authenticated session |
| Was the page loaded anyway? | No — the bot went and got it | Yes — you're looking at it |
| Scale | Millions of URLs, continuously | The pages a human actually opens |
| Discovers new URLs? | Yes, that's the point | No — you navigate, it reads |
| Costs the publisher a fetch | Yes, every time | No additional fetch beyond your own visit |
| Shows in the publisher's analytics as | A bot | A human visitor, because you are one |
ScrapeMaster sits firmly in the right-hand column. It opens in a side panel and reads the DOM of the page you already loaded. When you extract a table, the page has already been fetched, rendered, counted in the publisher's analytics, and shown you its ads. The extraction is a reading operation on bytes already sitting in your computer's memory. No second fetch. No crawl.
A rule blocking Training and Agent crawlers on ad-displaying pages targets the economic problem of bots consuming content without participating in the economics that fund it. If you loaded the page in your browser, you did participate. That's not a loophole; it's the actual distinction the rule is drawing.
Now the part where we don't oversell it
If we stopped there, this would be a comfortable post. It isn't that simple, and pretending otherwise would be exactly the kind of thing we wrote a manifesto to avoid.
The Agent category is aimed at browser-use agents
Cloudflare's Agent category is aimed at browser-use agents — chat-fetch bots and AI models driving Chrome. Read that again, because it's uncomfortable for anyone in the browser-extension business. "An AI model driving Chrome" is also an in-browser operation, in a real browser session, on a page that was really loaded. The distinction I just drew — datacenter vs. browser — doesn't cleanly separate ScrapeMaster from an Agent bot, because Agent bots increasingly live in browsers too.
What separates them isn't the runtime. It's who decided to load the page. With ScrapeMaster, a person navigated to a page and clicked a button. With a browser-use agent, a model decided to visit a URL as a step in a task, and may visit fifty more without a human seeing any of them. That's a real difference — it's the difference between a tool and an actor — but it is exactly the line the industry is now arguing about, and it is not a settled line. We're not going to tell you Cloudflare has definitively drawn it on our side of the fence, because Cloudflare hasn't said that and it isn't our call.
If you want the longer version of this argument, we've written about how ScrapeMaster differs from agentic browsers doing structured extraction, and about the Amazon–Perplexity injunction, which is the sharpest legal test of "is an agent a user?" we have so far.
Anti-bot systems have false positives
Bot detection is statistical. Request timing, mouse movement, TLS fingerprints, header ordering — a hundred signals in, a probability out. Probabilities are wrong sometimes. If you extract 400 rows across 20 paginated pages in 90 seconds, you look less like a reader and more like a script, because in that moment, behaviorally, you kind of are one. Real humans get CAPTCHA'd on Cloudflare-protected sites every day with no extension involved.
So: could ScrapeMaster usage trip an anti-bot system? Yes. Especially with pagination on an aggressive site, at speed, with no delays. That's not a hypothetical.
What we deliberately don't do about it
Here's our design position, stated plainly:
- We do not rotate proxies. Your requests come from your IP because they are your requests.
- We do not rotate or spoof fingerprints. Your browser looks like your browser.
- We cannot solve CAPTCHAs. Not "we don't by default" — we can't. There's no module for it.
- We cannot bypass logins or paywalls. If you can't see it, neither can the tool.
Every one of those is a capability we could build. Octoparse, ParseHub, and Import.io all sell proxy rotation as a headline feature; much of the commercial scraping industry is built on evasion infrastructure. We're not building it, and not because we ran out of time.
The reason is simple: the moment you add proxy rotation, you've stopped being a tool that reads pages you're looking at and started being a tool that defeats the site's decisions about who gets in. Those are different products with different ethics. We picked one. The cost is that if a site blocks you, we can't fix it. Configurable extraction delays are the only mitigation we offer, and they work by being more polite, not less detectable.
That's a deliberate position, not a gap we're hiding behind a roadmap.
Content-Usage isn't a Cloudflare land grab
A reasonable cynical read: the largest CDN on earth invented a syntax, made itself the enforcer, and started a payments business on top. Worth being suspicious about.
Except there's a standards track underneath it. The IETF AIPREF working group is standardizing exactly this layer:
draft-ietf-aipref-vocabdefines the vocabulary — the actual terms for expressing AI usage preferences, including yes/no signals fortrain-aiandsearch.draft-ietf-aipref-attachdefines how you attach those preferences: it specifies theContent-UsageHTTP header and the correspondingrobots.txtrule.
And draft-ietf-aipref-attach carries an August 2026 milestone to send a standards-track specification to the IESG. That's next month. So Content-Usage isn't a proprietary signal Cloudflare made up — it's an IETF work item that Cloudflare is shipping an implementation of, roughly in step with the spec's own timeline.
That matters practically: signals that go through the IETF tend to be honored by parties who never signed a contract with the CDN. robots.txt is respected far beyond anyone's ability to enforce it, because it became a norm. Content-Usage has a plausible path to the same status — and if it gets there, "what may you do with this content" becomes a machine-readable question with a standard answer. That's genuinely new.
The legal tension: robots.txt has teeth it doesn't legally have
Here's where it gets interesting, and where the two halves of this post collide.
In December 2025 — and please note that date carefully, it is 2025, not 2026 — Judge Sidney Stein held in Ziff Davis v. OpenAI that robots.txt is not a DMCA technological protection measure. His framing was that robots.txt is closer to a "keep off the grass" sign than a fence. It requests that you stay out; it doesn't control access. And because it doesn't control access, ignoring it isn't "circumvention" under the DMCA.
So as of that ruling, robots.txt is legally a polite notice. It's a sign on a lawn.
And yet: from September 15, 2026, a Content-Usage signal in that same robots.txt file will be backed by a CDN that will actually block your request at the edge if your bot's category doesn't match. The sign didn't become a fence. Somebody built a fence next to the sign.
That's the tension worth understanding, and it's the whole post:
| robots.txt (legal status) | robots.txt + Cloudflare enforcement (practical status) | |
|---|---|---|
| Is it a DMCA technological protection measure? | No (Stein, Dec 2025) | Still no — the law didn't change |
| Does ignoring it = circumvention? | No | Still no |
| Can you ignore it? | Legally, without DMCA liability | Practically, no — you get a 403 |
| What actually stops you? | Nothing but norms | Cloudflare's edge |
| Where the teeth come from | — | Commercial infrastructure, not statute |
The courts said robots.txt isn't a lock. The infrastructure layer responded by attaching a lock to it. Both things are now true, and they don't contradict each other — they just mean the enforcement of web crawling norms has moved from the legal system to the CDN layer, where it was always going to be more effective anyway.
If you want the full legal picture, we went deep on robots.txt and the DMCA scraping rulings, and on the line between AI training and commercial scraping.
What you should actually do
Practical, in order:
- Read the site's terms. Not the robots.txt — the terms. The Stein ruling is a reminder that contract claims live somewhere completely different from DMCA claims, and terms of service are enforceable regardless of what robots.txt says.
- Respect
Content-Usagesignals. Even though robots.txt isn't legally a lock, and even though you're not a crawler. A publisher sayingreferencerather thanfullis telling you something real about their intent. Honor it. - Don't hammer. Cloudflare's own >50% re-fetch number is the whole industry's shame in one statistic. Don't add to it. If you extracted a page yesterday and it hasn't changed, you don't need it again.
- Use extraction delays. They're configurable in ScrapeMaster. Use them. They cost you seconds and they cost the publisher nothing.
- Know where the actual risk is. Scraping publicly-accessible data is generally legal in most jurisdictions. That's not where people get hurt. People get hurt by re-publishing copyrighted content, violating terms of service they agreed to, and collecting personal data in scope of GDPR or CCPA. Those three are the real minefield. We wrote a whole guide on web scraping privacy compliance, and a general one on whether web scraping is legal. Neither is legal advice.
Frequently asked questions
Will Cloudflare's September 15 default block ScrapeMaster?
The default targets Training and Agent crawlers — automated fetchers — on ad-displaying pages. ScrapeMaster doesn't fetch pages; it reads pages you already loaded in your own browser session. There's no separate request for Cloudflare to categorize. That said, anti-bot systems are statistical and can produce false positives, especially during fast paginated extraction. We can't promise you'll never be challenged.
Does ScrapeMaster identify itself as a bot?
There's no bot user agent, because there's no separate fetch to attach one to. Your page loads are your browser's normal page loads. Follow detail opens links in background tabs — those are real browser navigations in your session, not synthetic requests.
Can ScrapeMaster get around a Cloudflare challenge?
No. We can't solve CAPTCHAs, don't rotate proxies, and don't spoof fingerprints. If Cloudflare challenges you, you solve it as yourself, in your browser, like any other visitor. If a site blocks you, it blocked you.
Is Content-Usage a real standard or a Cloudflare invention?
It's an IETF work item. draft-ietf-aipref-attach in the AIPREF working group defines the Content-Usage HTTP header and the robots.txt rule, with an August 2026 milestone to send a standards-track spec to the IESG. draft-ietf-aipref-vocab defines the vocabulary. Cloudflare is shipping an implementation, not inventing the syntax.
Since robots.txt isn't legally binding, can I ignore it?
The Ziff Davis v. OpenAI ruling (Dec 2025) held robots.txt isn't a DMCA technological protection measure, so ignoring it isn't DMCA circumvention. That's a narrow holding about one statute. It says nothing about breach of contract, copyright, or data protection — and from September 2026, Cloudflare will enforce at the edge regardless of what the DMCA says. "Not a DMCA violation" is a very low bar.
What's the difference between Pay Per Crawl and Pay Per Use?
Pay Per Crawl paid publishers when a bot fetched a page. Pay Per Use pays publishers when their content shapes an AI answer. Cloudflare's stated reason is that over 50% of AI crawl traffic is re-fetches of unchanged pages — so paying per fetch would mostly pay for waste. Ceramic.ai and You.com are launch partners.
How is this different from Octoparse or ParseHub?
Cloud scrapers like Octoparse, ParseHub, and Import.io run crawls from their own infrastructure with proxy rotation — which puts them squarely in the category Cloudflare's rules address. Browser extensions like Simplescraper, Web Scraper.io, Instant Data Scraper, Thunderbit, and ScrapeMaster read pages in your session instead. The category difference is real; whether a given site treats it as real is up to that site.
Where does my extracted data go?
Nowhere. It's stored locally in your browser's IndexedDB and exports to CSV, XLSX, JSON, or your clipboard. The only network request is during auto-detect, when the page's HTML structure — not its content — goes to our analysis API so it can suggest columns. Your extracted data is never uploaded.
Bottom line
September 15, 2026 is a real date, the default really does flip, and it really does cover all existing free Cloudflare customers. If you run bots, that matters enormously. If you read pages in a browser and pull structured data out of them, it matters much less than the headlines will suggest — because the rule is aimed at crawlers consuming content outside the economics that fund it, and you loaded the page.
But the line between "person with a tool" and "agent" is genuinely contested, Cloudflare's Agent category explicitly reaches toward browser-use agents, and false positives happen. So the honest posture is the same one we've always had: read the terms, respect Content-Usage, don't hammer, use delays, keep personal data out of it, and don't expect a tool to rescue you from a site that decided you're not welcome.
ScrapeMaster is free, runs in a side panel, auto-detects tables in 2–4 seconds, handles pagination and detail pages, stores everything locally, and cannot bypass a single thing. That last one isn't an apology. It's the design.