Cloudflare splits AI training from search indexing

Cloudflare has separated two things that used to arrive on the same crawler. From September 15, 2026, a site can refuse to have its content used for AI training while staying fully indexed in search. Cloudflare calls the setting "Disallow AI Training" and puts it in its security settings.

Until now that was one choice, not two. A crawler that gathered pages for a search index and for model training was a single visitor, so blocking the training also removed you from search. Cloudflare's press release puts the scale of that problem at 36.6% of verified crawler traffic on its network. The demand is broad: PYMNTS reports that nearly 90% of organizations call managing bot activity a challenge.

How the signal actually travels

The mechanism is deliberately boring, which is what makes it usable. Cloudflare publishes your preference in robots.txt through a feature it now calls Bot Preference Sync, replacing what was Managed Robots.txt. The setting itself replaces the older "Block AI Bots" switch.

Cloudflare's press release names robots.txt as the primary opt-out and says it also accepts comparable standards, including HTTP header preference signals from the emerging ai-prefs work at the IETF.

The controls split traffic three ways, each independently switchable: Search, AI Training and AI Agents. That third category is the one to think about, because an agent fetching your page on behalf of a user is neither indexing it nor training on it.

Enforcement has two halves. Crawlers Cloudflare considers accountable are expected to honor the preference. Mixed-use crawlers that do not separate their purposes get blocked outright when you turn restrictions on, and Cloudflare says it reports compliance through Radar.

Who has signed up, and what happens by default

The support list matters more than the feature, because a preference only works if crawlers read it.

Item	Detail
Accountable mixed-use crawlers	Apple, Google and Microsoft have met the criteria or committed to
Already separate search and training	Amazon, Anthropic, Meta and OpenAI
Timing	Apple and Google URL tools cited for early 2025; Microsoft robots.txt support targeted for early 2027
New ad-supported domains	AI Training and Agent blocked by default on pages carrying ads; Search allowed
New domains without ads	Search, Training and Agents all allowed by default
Existing domains	Settings migrate to preserve their current practical effect
Availability	All customers, on every plan tier

The default split is the detail to read twice. Cloudflare is drawing a line around ad-supported pages specifically, which is a statement about whose economics it thinks are at risk.

What this means for developers

If you run a site on Cloudflare, check which state you are in rather than assuming. Existing domains keep their effective behavior, new ones get defaults that depend on whether the domain shows ads, and the old "Block AI Bots" switch has been replaced. Those three facts mean your current setting is probably not what you would choose today.

Decide the agent question separately from the training question. Blocking AI Agents stops assistants from fetching your pages for a user who asked, which is closer to blocking a browser than to blocking a scraper. If your product expects people to reach it through an assistant, that toggle cuts a path you may want open.

Do not treat robots.txt as enforcement. It is a request that accountable crawlers honor, and Cloudflare's own framing splits the world into crawlers that respect preferences and ones it blocks by force. The blocking is the part with teeth, and it only applies to traffic Cloudflare can identify.

Watch the timelines before you rely on this. Microsoft robots.txt support is targeted for early 2027, which means the promise and the plumbing are not landing together. This is the same argument as Cloudflare's Gateway feature that detects MCP traffic. The network layer is deciding what an AI system may do with your content.

This article was first published on Tech AI Wire.