Should I block AI crawlers like GPTBot?
Block training if you want to, and keep search open. OpenAI runs separate crawlers: GPTBot for training its models and OAI-SearchBot for ChatGPT search, and sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers.” For Google, AI Overviews follow Googlebot; Google-Extended covers AI training and grounding in some other Google systems.
Key figure
2
OpenAI crawlers with different jobs: OAI-SearchBot finds pages for ChatGPT search, GPTBot collects pages for training.
The full answer.
| Crawler | What it does | If you block it |
|---|---|---|
| OAI-SearchBot (OpenAI) | Finds pages for ChatGPT search answers. | Your site is not shown in ChatGPT search answers (Q2). |
| GPTBot (OpenAI) | Collects pages that may train OpenAI’s models. | Training use stops; ChatGPT search is unaffected (Q2). |
| ChatGPT-User (OpenAI) | Visits a page when a user asks ChatGPT to. | OpenAI says robots.txt rules “may not apply” (Q2). |
| Googlebot (Google) | Crawls for Search, including AI Overviews and AI Mode. | You leave Google Search, AI features included (Q1). |
| Google-Extended (Google) | A robots.txt setting for AI training and grounding in some other Google systems. | Google Search is unaffected (Q1). |
Check your host too. A firewall or CDN can turn crawlers away even when robots.txt lets them in. Google lists crawling allowed “in robots.txt, and by any CDN or hosting infrastructure” among the basics (Q1). Anthropic and Perplexity publish their own crawler names; read their pages before writing rules for them.
What Still Fox does.
The free snapshot checks whether each AI crawler can reach your site, using each crawler’s name.
Sources.
Each claim above carries its id and a grade from A to D. The research report holds the full ledger.
- Q1 EC
Google Search Central, AI features and your website
There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.
You don’t need to create new machine readable files, AI text files, or markup to appear in these features.
Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure
Checking that your Merchant Center and Business Profile information is up-to-date
- Q2 EC
OpenAI, Overview of OpenAI Crawlers
Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers
Because these actions are initiated by a user, robots.txt rules may not apply.
More answers.
- How do I get my business recommended by ChatGPT?→
- How do I check what ChatGPT says about my business?→
- What is AI visibility, and is it just SEO with a new name?→
- Are Google’s AI Overviews taking traffic from my website?→
- Why does AI get my business hours wrong, or say I’m closed?→
- Does my website need an llms.txt file?→