If Cloudflare is blocking ChatGPT, go to Cloudflare's security settings and set the Search and Agent AI policies to Allow. Then remove any AI Crawl Control block, old WAF custom rule or Bot Fight Mode action that still stops OpenAI's OAI-SearchBot and ChatGPT-User, check that robots.txt allows both, and test from outside your network. You can keep GPTBot, OpenAI's training crawler, blocked if you like. ChatGPT search doesn't use it.
Why is Cloudflare blocking AI crawlers like ChatGPT?
Because at some point someone on the account chose to block them, maybe last month, maybe years ago. That block can sit in any of five places: AI access policies, AI Crawl Control, WAF custom rules, Bot Fight Mode or the managed robots.txt. Fix one and leave another, and the bot still gets turned away.
Cloudflare added a one-click option to block AI crawlers in September 2024, and more than one million customers turned it on. On 1 July 2025 it went further. Every new domain is now asked up front whether to allow AI crawlers, and the press called it "block by default".
The settings have moved around a few times since, so a lot of older guides point at toggles that aren't there any more. Here's where a block can live today:
- AI Crawler and Agent access policies: three policies (Search, Agent, Training) under Security settings.
- AI Crawl Control: a per-crawler Allow or Block list. A block here creates or updates a WAF custom rule on your zone.
- Your own WAF custom rules: an old "block all bots" rule can quietly override everything else.
- Bot Fight Mode: the free-plan bot protection, which runs outside the normal rules engine.
- Managed robots.txt: Cloudflare can add
Disallowlines above your own rules.
According to a recent write-up by NotHuman, some newer domains now get Search set to Allow by default. So if you're blocked now, it's almost always an old choice or an old rule still hanging around, not a new default.
Which OpenAI bots need access for ChatGPT to see your site?
Two of them: OAI-SearchBot for ChatGPT search results and ChatGPT-User for pages a user asks about. GPTBot only collects training data. OpenAI splits these jobs across separate bots, and you control each one on its own (OpenAI's crawler documentation). That split is what lets you say yes to search and no to training.
| Bot | What it does | Cloudflare policy | Allow it? |
|---|---|---|---|
| OAI-SearchBot | Finds and indexes pages for ChatGPT search results | Search | Yes, if you want to be cited |
| ChatGPT-User | Fetches a page when a user asks ChatGPT about it | Agent | Yes |
| GPTBot | Collects content that may be used to train models | Training | Your choice |
| Claude-SearchBot, Claude-User | Anthropic's search and user fetchers | Search, Agent | Yes, for Claude answers |
| ClaudeBot | Anthropic's training crawler | Training | Your choice |
| PerplexityBot, Perplexity-User | Perplexity search and user fetches | Not a verified bot on Cloudflare | Needs an IP-based rule |
The policy mapping comes from the NotHuman guide. Perplexity is the awkward one. Cloudflare delisted it as a verified bot in August 2025, so a "known bots" exception won't cover it.
How do you let ChatGPT in on Cloudflare, step by step?
Go through five places in Cloudflare, in this order: the AI access policies, the AI Crawl Control list, your WAF custom rules, Bot Fight Mode and the robots.txt Cloudflare actually serves. Then test from outside your network.
- 1
Set the AI policies
Security settings, filter by Bot traffic, open AI Crawler and Agent access policies. Set Search and Agent to Allow, and Training to Disallow or Allow.
- 2
Check AI Crawl Control
In AI Crawl Control, open the Security tab and set OAI-SearchBot and ChatGPT-User to Allow.
- 3
Audit WAF custom rules
Look for old rules that block or challenge bots, AI crawlers or empty user agents, and add a Skip rule for verified AI search bots.
- 4
Check Bot Fight Mode
Look in Security events for blocks where the service is Bot Fight Mode. It cannot be bypassed with a WAF rule.
- 5
Fix robots.txt
Open yourdomain.com/robots.txt in a browser and make sure nothing above your own rules disallows OpenAI's search bots.
- 6
Test from outside
Run the AI Crawler Checker and look at the AI Crawl Control Crawlers tab over the next few days.
1. Set the AI Crawler and Agent access policies
In the Cloudflare dashboard, go to Security settings, filter by Bot traffic and open AI Crawler and Agent access policies. Set Search to Allow and Agent to Allow. For Training, pick Disallow if you don't want your content used to train models, or Allow if you're fine with it.
Cloudflare's robots.txt documentation says that with Bot Preference Sync on, Training set to Disallow adds the robots.txt lines and also blocks training crawlers at the edge, while mixed-use crawlers are left out of that block. I'd stick with Disallow unless you have a specific reason.
2. Check the AI Crawl Control list
Go to AI Crawl Control and open the Security tab. You'll see every AI crawler that has visited, each with an action. Set OAI-SearchBot and ChatGPT-User to Allow. Don't skip this one. Each Block here becomes a WAF rule, and those rules stay put even after you change the policies in step 1.
On the free plan, AI Crawl Control identifies crawlers by user agent string.
3. Audit your WAF custom rules
Go to Security rules and read every custom rule. You're looking for anything with cf.client.bot, a verified bot category, "AI", or a broad user agent match set to Block or Managed Challenge. Delete or tighten anything you don't need any more.
If you want a belt-and-braces exception, add a custom rule with the Skip action and put it where it runs before your blocking rules:
cf.client.bot and (
http.user_agent contains "OAI-SearchBot" or
http.user_agent contains "ChatGPT-User" or
http.user_agent contains "Claude-SearchBot" or
http.user_agent contains "Claude-User"
)
cf.client.bot only matches bots Cloudflare has verified, so a scraper that copies the user agent name won't slip through. Don't use cf.client.bot by itself, though. GPTBot and ClaudeBot are verified too, so a broad skip lets the training bots past your block as well.
4. Check Bot Fight Mode
This is the one that catches people out. Cloudflare's docs say you cannot bypass Bot Fight Mode with WAF custom rules or Page Rules. It runs in a separate pipeline where Skip, Bypass and Allow simply do nothing.
A Skip rule won't fix Bot Fight Mode
If Security events show Bot Fight Mode acting on OpenAI's bots, your options are to turn it off, add IP Access rules that match first, or move to a paid plan with Super Bot Fight Mode, which supports Skip rules.
Open Security events, filter by the user agent and look at the Service field. If it says Bot Fight Mode, you've found your blocker.
5. Check what robots.txt actually serves
Open https://yourdomain.com/robots.txt in a browser. If there's a block starting # BEGIN Cloudflare Managed content, Cloudflare is adding its own directives above your file. That block only shows up when Bot Preference Sync is on and at least one policy is restrictive. Set every policy to Allow and your file is served as you wrote it. If you keep Training on Disallow, the managed block stays and already disallows the training bots, so you can drop GPTBot, ClaudeBot and CCBot from your own file to avoid duplicate groups.
What should robots.txt say for each AI bot?
Allow the AI search and user-fetch bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User). Disallow the training bots (GPTBot, ClaudeBot, CCBot) only if you want out of training. This is the starting point I'd use. It does exactly that and lets everything else crawl as normal:
# AI search and user-triggered fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# AI training crawlers: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xml
If you're happy for your content to be used in training, move GPTBot into the allowed group. The User-agent: * lines are for WordPress, so change them to suit your platform.
Two things to watch. First, don't write an Allow for a bot that Cloudflare's managed block also names. Crawlers merge duplicate groups in different ways, so you can't predict what happens. Change the policy in Cloudflare so the managed block disappears, instead of trying to override it. If you keep Training on Disallow in Cloudflare, the managed block stays and already disallows the training bots, so drop the GPTBot, ClaudeBot and CCBot group from your own file to avoid duplicate groups. Second, robots.txt is a request, not a lock. Cloudflare's own docs say compliance is voluntary, and an Allow line is no use if the Cloudflare edge is still returning 403.
Sometimes the managed block sticks around after you've changed the settings. There's a Cloudflare Community thread about exactly that. While you're at it, check whether an SEO plugin, your host or a physical robots.txt file on the server is serving rules you thought you'd removed.
Do you need llms.txt as well?
Not for access. llms.txt doesn't allow or block anything. It's a plain Markdown file at /llms.txt that gives AI tools a short, clean summary of your site with links to your key pages. robots.txt decides whether a bot may fetch a page; llms.txt helps it make sense of what it fetched. Cloudflare publishes one for its own docs, and I keep one on mohitgoyal.in so AI assistants can answer questions about my work without running JavaScript.
Once access is sorted, add one. I've written up the format and setup in what llms.txt is and how to add one.
How do you check that ChatGPT can reach your site?
Test in four places: a robots.txt checker, a curl request with the OAI-SearchBot user agent, the AI Crawl Control Crawlers view and Cloudflare's Security events. A request has to get past every layer, so one passing test doesn't prove much on its own.
- Run the AI Crawler Checker. I built it to read your live robots.txt and tell you whether 14 AI crawlers, including OpenAI's, Anthropic's, Perplexity's and Google's, may fetch your home page, and which rule decides it. It also checks your llms.txt and lists the sitemaps your robots.txt declares. It's free and uses no AI.
- Fetch the page with a bot user agent. In a terminal, run
curl -I -A "OAI-SearchBot" https://yourdomain.com/. A 403 or a challenge page means something at the Cloudflare edge is blocking by user agent. A 200 is a good sign but not proof, because your curl request isn't a verified bot. - Watch AI Crawl Control. The Crawlers view shows allowed and unsuccessful requests for each crawler. After the fix, you should see OAI-SearchBot and ChatGPT-User requests start getting through.
- Read Security events. Filter by user agent and see which rule or service acted on any blocked request.
Should you allow GPTBot too?
GPTBot makes no difference to ChatGPT search citations either way (OpenAI's crawler documentation). It only decides whether OpenAI may use your content to train models. So this is a business decision, not a visibility fix.
| Choice | Allow GPTBot | Block GPTBot |
|---|---|---|
| ChatGPT search citations | Not affected | Not affected |
| Content used for model training | Possible | Opted out |
| Best for | Brands that want maximum AI exposure | Publishers protecting paid or original content |
For most of the business sites I work on (service pages, case studies, product catalogues), being seen is worth more than keeping content out of training. A publisher might reasonably decide the other way. The part that matters is letting the search and agent bots in, because they're the ones that put your pages into ChatGPT answers. Access alone won't get you cited, though. I go through structure, schema and content in how to get cited in AI Overviews and ChatGPT.
What if you can't find the setting that's blocking it?
If nothing in Cloudflare explains it, look at the origin server. Security plugins, host-level firewalls and old .htaccess rules can all send 403s to bot user agents. Server-level rules are easy to forget because they sit outside Cloudflare's dashboard.
If you've just moved the site onto Cloudflare, or someone else set the account up years ago, audit the whole zone in one pass: DNS, SSL mode, WAF rules, bot settings and caching. I do that as part of hosting migrations and maintenance.
Start by running your domain through the AI Crawler Checker. If OpenAI's bots still can't get in, or you'd rather have robots.txt, llms.txt, schema and Cloudflare set up properly in one go, take a look at my technical SEO service or book a 20-minute call and send me the domain.
