# How to block AI crawlers without dropping out of AI search

> Disallow the training bots, keep the search bots, check the live file, then watch the logs.

URL: https://claudegpt.si/news/block-ai-crawlers-keep-ai-search-guide  
Published: 2026-10-26T15:55:47.627Z  
Author: ClaudeGPT Si Newsroom  
Section: Guides  
Topics: AI crawlers, Robots.txt, GPTBot  
Publisher: ClaudeGPT Si

## Key takeaways

- OpenAI, Anthropic and Perplexity run separate training, search and user-requested bots, so a site can block training without leaving AI search.
- Robots.txt does not bind user-requested fetchers: OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
- The seogeoaeo.ai robots.txt and AI crawler checker confirms which bots a live file blocks for 10 credits a run, and its log analyzer shows which still visit.

To block AI crawlers without dropping out of AI search, disallow the training bots (GPTBot, ClaudeBot, Google-Extended) in robots.txt and leave the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) open. Then confirm the result with an AI crawler check such as [seogeoaeo.ai](https://seogeoaeo.ai), because one misplaced group can block more, or less, than you meant.

## Decide what you are blocking

OpenAI, Anthropic, Perplexity and Google each split their bots by job. The split decides what a block costs you.

| Vendor | Training bot | Search bot | User-requested fetcher |
| --- | --- | --- | --- |
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | none listed | PerplexityBot | Perplexity-User |
| Google | Google-Extended (a token) | Googlebot (Google Search) | none relevant here |

Blocking a training bot is a signal that your content should stay out of model training. Blocking a search bot is a different decision: OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and Anthropic warns that disabling Claude-SearchBot may reduce your visibility in its search results. Perplexity says PerplexityBot is not used for model training at all.

Google-Extended deserves a note. It has no user agent of its own, and Google says it does not affect inclusion or ranking in Google Search. It controls use of your content for Gemini training and for grounding in Gemini Apps and Vertex AI.

## Step 1: write the rules

A training-only block looks like this:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

To block everything AI-related, add groups for OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot as well, knowing you trade away AI search citations.

Two rules trip people up. A crawler follows only the most specific group that names it, so these new groups replace your `User-agent: *` rules for those bots rather than adding to them. And robots.txt only covers the host it sits on; Anthropic asks site owners to repeat the block on every subdomain they want to opt out.

## Step 2: know what robots.txt cannot do

Robots.txt is a request, not a lock. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. If you need those blocked too, that means firewall or CDN rules.

Be careful with IP blocks. Anthropic notes that blocking its IPs may not work as a reliable opt-out, because it stops ClaudeBot from reading your robots.txt. OpenAI and Perplexity publish IP lists for their bots, which helps you allow the real ones and spot impostors.

## Step 3: check the live file

Once the new file is deployed, run the seogeoaeo.ai [Robots.txt & AI Crawler Checker](https://seogeoaeo.ai/tools/robots-checker). It lists every rule that blocks a search or AI crawler, shows access for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, and names the line to change. A run costs 10 credits, shown before it starts. Test a few paths as `GPTBot` and as `OAI-SearchBot` to be sure the first is blocked and the second is not.

## Step 4: watch what happens

Changes are not instant. OpenAI says ChatGPT search can take about 24 hours to adjust, and Perplexity gives up to 24 hours. After that, paste a slice of your access log into the seogeoaeo.ai AI Bot Log Analyzer (8 credits) to see which AI bots still visit, which requests were refused, and which visits used a bot's name from an IP its operator does not publish.

## When to stop short

If your business depends on being recommended by ChatGPT, Claude or Perplexity, blocking their search bots works against you. If you want citations but not training, the training-only block above is the one to use. And if you only want bots to slow down, Anthropic supports a `Crawl-delay` line for ClaudeBot, which is gentler than a full block.

## Frequently asked questions

### Will blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended does not, according to Google: it does not affect inclusion or ranking in Google Search. Blocking Googlebot itself would, so leave it alone.

### Can I block AI crawlers by IP address instead?

You can, but Anthropic warns that blocking its IPs may not work as a reliable opt-out because it stops ClaudeBot reading robots.txt. Robots.txt is the method the vendors document.

### How long until a block takes effect?

OpenAI says ChatGPT search can take about 24 hours to adjust after a robots.txt change, and Perplexity says up to 24 hours.

## Sources

- [Robots.txt & AI Crawler Checker](https://seogeoaeo.ai/tools/robots-checker), seogeoaeo.ai
- [AI Bot Log Analyzer](https://seogeoaeo.ai/tools/ai-bot-log-analyzer), seogeoaeo.ai
- [Overview of OpenAI crawlers](https://platform.openai.com/docs/bots), OpenAI
- [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots), Perplexity
- [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), Google