Cloudflare launched a new tool called Bot Preference Sync to help website owners control AI crawlers. According to the Cloudflare Blog (published August 21, 2026, and updated August 22, 2026), the feature writes your dashboard AI choices directly into your site’s robots.txt file.
How the new sync works
Instead of editing text files by hand, you pick settings in your Cloudflare dashboard. The sync automatically adds generated rules to the top of your existing robots.txt file.
The tool groups bot traffic into three clear buckets:
- Search: Choose Allow, Block on pages that serve ads, or Block everywhere.
- Agent: Choose Allow, Block on pages with ads, or Block everywhere.
- Training: Choose Disallow to tell AI builders not to scrape your data for model training.
Existing Disallow lines stay in your file. You can turn the sync off whenever you want. For new customers, Cloudflare turns sync on by default. Publisher onboarding lets you pick a Training Disallow default right away, while standard onboarding starts without blocks.
Cloudflare positions the feature for Free through Enterprise plans, with rollout expanding across zones.
What you shouldn’t assume yet
These claims come straight from Cloudflare’s announcement — no independent third party has verified crawler compliance, traffic impact, or security gains. Keep in mind that robots.txt is only a request. It does not force bad bots to obey. Blocking edge requests also does not prove that companies will stop using previously scraped data.
Cloudflare also outlined its wish list for crawler transparency: respecting no-training tags, letting sites opt out of AI summaries, and providing URL-level traffic metrics. Those are Cloudflare’s policy goals, not industry rules that every bot currently follows.
Why this matters for site owners
Hand-crafting bot rules is messy. Every new AI startup brings another crawler name. Bot Preference Sync gives you a single set of switches for Search, Agent, and Training bots.
There is a trade-off. The sync does not read complex custom rules. If you run special path filters, you must check that the generated top block does not clash with your setup.
Test it on a staging site
Don’t flip the switch on your main domain right away. Try this instead:
- Download your current
robots.txtfile from a test zone. - Turn on Bot Preference Sync. Set Search to Allow, Agent to Allow, and Training to Disallow.
- Download
robots.txtagain to check that your old custom lines are still intact. - Check your edge logs to see how Cloudflare classifies incoming crawlers.
It is a helpful shortcut. Just remember that a line of text cannot replace active log monitoring.