AI in Action
AI agent crawlers now need permission. Here’s how to get it
Dashveenjit Kaur
July 13, 2026
Share this story:
Tags:
agentic ai
AI agent crawlers
Cloudflare
content licensing
Matthew Prince
publishers
web crawlers
Categories:
AI Business Strategy
AI in Action
Artificial Intelligence
Inside AI
AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. Cloudflare
announced
the change on July 1, and most of the coverage since then has focused on Google. The more useful part is what it asks of everyone building agents, and what it offers them in return.
Cloudflare has replaced its single block-AI-bots switch with three categories. Search covers bots that index a page to answer questions about it later. Agent covers automated systems acting in real time for a user, including ChatGPT’s fetch bot and browser-driving agents. Training covers crawlers that pull content into a model’s weights. The controls went live on July 1 for every customer, including the free tier.
From September 15, the defaults change. Training and Agent will be blocked on pages that display ads, while Search will stay allowed. The new defaults apply to domains newly onboarding to Cloudflare, new sites set up by existing customers, and all existing free-tier customers.
Anyone who does not want them can opt out through their security settings before the date.
Cloudflare’s logic is that an advertisement is evidence that a page was built for a human to land on. A search crawler that sends a reader back is a referral. A bot that reads the page and hands the answer to someone else is not.
What AI agent crawlers run into now
Agentic deployments have been built on the assumption that the open web stays open. A research agent fetches a competitor’s pricing page. A monitoring tool checks a supplier’s announcements. A customer-service agent pulls a manufacturer’s specification sheet. None of this involves a licence, and until now, none of it needed one.
Cloudflare sits in front of a large share of the world’s web traffic, and its blocks operate at the network level rather than as a robots.txt suggestion a crawler can ignore. Ad-supported pages are exactly the pages agents want, because that is where news, reviews, pricing and product coverage live. The failure mode for an enterprise agent is not a lawsuit. It is silence, or an answer built from whatever it could still reach.
There is a Google-shaped complication. Googlebot crawls for both search and training in a single bot, so under the most restrictive rule, a site that blocks Training also blocks Googlebot. Cloudflare CEO Matthew Prince said the company hopes the changes will “encourage mixed-use crawlers to separate search from agent use and training”, which is a polite way of saying the pressure is the point.
Getting permission for AI agent crawlers
Anyone running agents should start by working out which of their Cloudflare accounts will read as Agent-class. The classification is behavioural rather than something you opt into, so a research agent that browses in real time is caught whether or not its operator thinks of it as a crawler.
Expect degraded coverage rather than a clean failure, because the block lands on ad-supported pages and leaves the rest reachable. Negotiated access, not a rewritten user-agent string, is the way through.
Publishers have a different homework list. Check your tier first, since existing free-tier customers are moved to the new defaults automatically on September 15, a detail most coverage skipped. Then decide whether blocking Training is worth what it costs, because it takes Googlebot with it and your search visibility along with it.
The mechanism worth watching is the money. Pay Per Crawl is becoming Pay Per Use, with Ceramic.ai paying publishers when their content appears in AI search results, and You.com paying when an agent reaches premium content. Cloudflare says more than half of AI crawler traffic is spent re-fetching pages that have not changed, so there is waste on both sides worth pricing out.
This is the first round of the content fight, where the answer on offer is a rate rather than a wall.
One weakness lies in the taxonomy itself. Search, Agent and Training are behaviours the AI companies declare about their own bots, and a firm that would rather not have its training run classified as training has an obvious incentive. The announcement does not explain what stops it.
Access to the open web has been free and unlimited for thirty years, and the bill is now itemised. Agent builders who sort out their access before September have a workable problem. The ones who find out from a 403 will be rebuilding on the fly.
See also:
Visa ChatGPT integration enables AI agent retail purchasing
Want to learn more about AI and big data from industry leaders?
Check out
AI & Big Data Expo
taking place in Amsterdam, California, and London. The comprehensive event is part of
TechEx
and is co-located with other leading technology events including the
Cyber Security & Cloud Expo
. Click
here
for more information.
AI News is powered by
TechForge Media
. Explore other upcoming enterprise technology events and webinars
here
.
About the Author
Dashveenjit Kaur
Journalist
Dashveenjit is an experienced tech and business journalist with a determination to find and produce stories for online and print daily. She is also an experienced parliament reporter with occasional pursuits in the lifestyle and art industries.
Related
How to shrink the token budget without shrinking the team
July 10, 2026
AWS GraphRAG deployment cuts drug research cycles by 87%
July 9, 2026
NHS AI blood test could reduce invasive womb cancer checks
July 9, 2026
Insilico Medicine advances AI drug for IPF to Phase III trials
July 7, 2026
Join our Community
Subscribe now to get all our premium content and latest tech news delivered straight to your inbox
Click here
Popular
Physical AI
Asylon and Thrive Logic bring physical AI to enterprise perimeter security
7906 view(s)
AI Business Strategy
,
Artificial Intelligence
,
Features
,
Finance AI
,
World of Work
JPMorgan begins tracking how employees use AI at work
7522 view(s)
AI in Action
,
AI Market Trends
,
Artificial Intelligence
,
Human-AI Relationships
,
Inside AI
,
Manufacturing & Engineering AI
,
Physical AI
Google makes its industrial robotics AI play official–and this time, it means business
6451 view(s)
Multimodal AI
Citizen developers now have their own Wingman
6384 view(s)
See all
Latest
View All Latest
Healthcare & Wellness AI
July 9, 2026
NHS AI blood test could reduce invasive womb cancer checks
Healthcare & Wellness AI
July 7, 2026
Insilico Medicine advances AI drug for IPF to Phase III trials
AI in Action
July 7, 2026
L’Oreal, Mondelez, and Nestle use AI to speed product development
