Guides AI search and AI visibility
Should you block AI crawlers?
What AI crawlers do, what 13 major news publishers actually block, and why the wrong rules can take you out of ChatGPT, Claude and Perplexity answers.
Most publishers and brands I talk to have made a decision about AI crawlers without quite meaning to. Someone added a few lines to robots.txt in 2023, or a CDN switched on a default, and nobody has looked since.
That matters because “AI crawlers” aren’t one thing. Some collect your pages to train AI models. Others decide whether ChatGPT, Claude or Perplexity can find and cite you when someone asks a question. Blocking the first is a licensing choice. Blocking the second can quietly remove you from AI answers.
What are AI crawlers?
An AI crawler is a bot that collects web pages for AI. There are three kinds:
| Job | What it does | Examples |
|---|---|---|
| Training | Collects pages that may be used to train AI models | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent |
| AI search | Builds the index that AI assistants search and cite | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User fetch | Opens a page when a person asks an assistant about it | ChatGPT-User, Claude-User, Perplexity-User |
The fourth group aren’t crawlers at all. Google-Extended and Applebot-Extended are switches. Google and Apple crawl your site with their normal bots. Applebot-Extended tells Apple whether it may train its AI models on what it collected. Google-Extended tells Google whether it may use your pages to train Gemini and to support answers in the Gemini app.
The AI bots publishers and brands ask about most
| Bot | Company | Job |
|---|---|---|
| GPTBot | OpenAI | Training |
| OAI-SearchBot | OpenAI | AI search (ChatGPT search) |
| ChatGPT-User | OpenAI | User fetch |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | AI search |
| Claude-User | Anthropic | User fetch |
| PerplexityBot | Perplexity | AI search |
| Perplexity-User | Perplexity | User fetch |
| Google-Extended | AI training switch | |
| Applebot-Extended | Apple | AI training switch |
| CCBot | Common Crawl | Training (public archive) |
| Meta-ExternalAgent | Meta | Training |
| Bytespider | ByteDance | Training |
The main AI companies document their bots: OpenAI, Anthropic, Perplexity, Google and Common Crawl. The details change often. New bots appear, old names are retired, and the companies revise what each one is used for. I check these pages every week.
What do news publishers actually block?
On 1 November 2026 I checked the robots.txt files of 15 major UK and US news publishers with my Robots.txt Checker. Two turned the checker away with bot protection, so the figures below cover the other 13.
| Bot | Job | Blocked from the whole site | Blocked from most of the site | Total (of 13) |
|---|---|---|---|---|
| Googlebot | Search | 0 | 0 | 0 |
| Bingbot | Search | 0 | 0 | 0 |
| ClaudeBot | Training | 6 | 4 | 10 |
| CCBot | Training | 6 | 4 | 10 |
| PerplexityBot | AI search | 6 | 4 | 10 |
| GPTBot | Training | 3 | 3 | 6 |
| Claude-SearchBot | AI search | 3 | 3 | 6 |
| Claude-User | User fetch | 3 | 3 | 6 |
| Google-Extended | AI training switch | 2 | 3 | 5 |
| OAI-SearchBot | AI search | 2 | 2 | 4 |
| ChatGPT-User | User fetch | 1 | 2 | 3 |
Two caveats. This is a small sample of big publishers, so treat it as a snapshot of practice, not a rule. And robots.txt is only part of the picture. Publishers can also block bots at their CDN or server, which a robots.txt check can’t see. The two sites that turned my checker away show that some do.
Three things stand out.
Nobody blocks Google or Bing. Every publisher keeps its main search crawlers in. They still send most search traffic, and Bing’s index also feeds Microsoft Copilot.
Publishers treat AI companies very differently. ClaudeBot is blocked by 10 publishers, but GPTBot by only 6 and OAI-SearchBot by 4. Several let OpenAI in while keeping Anthropic and Perplexity out. Some of that may reflect licensing deals. Some may just be rules added at different times as new bots appeared, and never reviewed.
Search bots get caught in “block AI” rules. Ten of the 13 block PerplexityBot and six block Claude-SearchBot. Those bots don’t train models. They decide whether Perplexity and Claude can cite you. Some publishers will have chosen that. Others may not know, because a broad “block AI” setting catches search bots too. It happened on my own site, as I explain below.
Does blocking AI crawlers affect Google Search or AI Overviews?
Blocking the AI companies’ bots doesn’t change how Google treats you. Google’s AI Overviews and AI Mode are part of Google Search and use Googlebot, and Google’s documentation says Googlebot’s rules are the control for Search, including its AI features. Google-Extended doesn’t remove you from AI Overviews either.
Google does offer ways to limit what appears in its AI features. But they also limit how your pages show in normal search results, for example by removing the description under your listing. There’s no clean “keep me out of AI Overviews but keep my rankings” switch.
What robots.txt can’t do
It’s a request, not a lock. Well-behaved crawlers follow it. Nothing forces them to.
Some bots say it may not apply to them. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page.
It doesn’t remove what’s already collected. Blocking a training bot today keeps your future pages out. It doesn’t pull pages out of models or archives that already have them.
One rule can undo another. A bot that has its own section in robots.txt ignores the rules you wrote for all bots. Adding a line to let one AI bot in can let it into pages you meant to keep closed.
Your CDN may be blocking bots without telling you. Cloudflare started blocking AI crawlers by default for new sites in July 2025, and it can add its own rules to your robots.txt. It happened on my own site. In September 2026 an audit showed Cloudflare’s “Block AI bots” setting was turning away GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User and PerplexityBot. So it was blocking the bots that let ChatGPT, Claude and Perplexity cite me, not just the training bots. It had also written AI rules into my robots.txt that I hadn’t chosen. Your robots.txt file can say one thing while your CDN does another.
What the right answer depends on
There isn’t a single right answer. Where you should land depends on how you make money, whether you’re talking to AI companies, how much AI assistants already matter to your audience, and what your CDN is doing behind the scenes. Blocking too much can remove you from the answers your audience is starting to rely on. Blocking too little gives your content away for nothing.
A new shared “no AI training” signal arrived in October 2026. Anthropic already honours it, Microsoft will from February 2027 and Google plans to in early 2027. It changes the options, but it interacts with existing rules in ways that are easy to get wrong.
How I help
I review what your site actually lets in and keeps out, across robots.txt, your CDN and your server, and what that’s costing or saving you in AI answers. You come away with a written policy your editorial, commercial and technical teams agree on, and the exact changes to make.
If you just want to see what your file says today, run your domain through the Robots.txt Checker. To get it right, that’s part of my AI visibility consulting.
Not sure what your robots.txt or your CDN is blocking, or what it's costing you in AI answers? I'll look at it with you on a free 30-minute call.
Got a question first? See the questions clients ask me.
Explore services