Key takeaways
- GPTBot is OpenAI's training crawler. Blocking it in robots.txt opts your pages out of model training, not out of ChatGPT search.
- Search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether you can be cited. Most businesses should allow them.
- User fetchers such as ChatGPT-User, Perplexity-User and Google-Agent act when a person asks, and may not follow robots.txt.
- The most common accident is not robots.txt at all: a CDN or firewall that blocks bots, including Googlebot.
A GPTBot rule in robots.txt is a business decision, not a technical default. If robots.txt itself is new to you, read my robots.txt guide first; this page only covers the AI crawlers. And note that llms.txt is a different file that blocks nothing, which I explain in what llms.txt is.
What are AI crawlers and what do they do?
AI crawlers are bots that AI companies send to read web pages. They do three jobs: collecting pages to train models, indexing pages so an AI search tool can cite them, and fetching a page live because a user asked about it. Each job usually has its own user agent name, so you can treat them differently in robots.txt.
Every name below comes from the company's own documentation, checked on 9 November 2026: OpenAI, Anthropic, Perplexity, Google's common crawlers and Google's user-triggered fetchers.
| Company | Training | Search and citations | User fetch (a person asked) |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User (OpenAI says robots.txt rules may not apply) |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None: PerplexityBot is not used for training | PerplexityBot | Perplexity-User (generally ignores robots.txt) |
| Google-Extended, a control token with no crawler of its own | Googlebot (AI Overviews and AI Mode use Search) | Google-Agent (user-triggered fetchers generally ignore robots.txt) |
OpenAI also lists OAI-AdsBot, which only visits pages submitted as ads in ChatGPT. Google says Google-Extended controls whether its crawled content may be used to train future Gemini models and for grounding in Gemini Apps, and that it does not affect inclusion or ranking in Google Search. For Search itself, Google's AI features page says the control is robots.txt for Googlebot.
Should you block GPTBot in robots.txt?
Only if you do not want your content used to train OpenAI's models. OpenAI says disallowing GPTBot signals exactly that, and search visibility is handled by a separate bot, OAI-SearchBot. For a service business that wants to be recommended, blocking GPTBot rarely helps; blocking OAI-SearchBot would remove you from ChatGPT search answers.
The same logic applies to ClaudeBot and Google-Extended. Blocking Google-Extended also covers grounding in Gemini Apps, so a business that wants Gemini to quote it should think twice. Publishers with paid content often block training bots; most local businesses, clinics and stores I work with want the opposite, to be read and cited as widely as possible.
Safe robots.txt examples for AI crawlers
Example 1, block training and keep search (a hypothetical store with a private /cart/ folder):
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Disallow: /cart/
User-agent: Claude-SearchBot
Allow: /
Disallow: /cart/
User-agent: PerplexityBot
Allow: /
Disallow: /cart/
User-agent: *
Disallow: /cart/
Notice that /cart/ is repeated in every group. Google's robots.txt specification says a crawler follows only the group with the most specific user agent that matches it, and that specific groups and the * group are not combined. A bot named in its own group ignores your general rules. Example 2, allow everything: if you want every AI tool to read you, you do not need any AI lines at all, as long as nothing in a * group blocks them.
- Copy the exact user agent names from the table; a typo means the rule does nothing.
- Never put Disallow: / under User-agent: * unless you want to vanish from search.
- Keep private pages out with logins, not robots.txt. It is a public file and it is not security.
- Allow about 24 hours for OpenAI and Perplexity to reflect a change, as both say in their docs.
Does blocking AI crawlers in robots.txt actually work?
For the declared crawlers of the large companies, mostly yes: OpenAI, Anthropic, Perplexity and Google document that their automatic crawlers follow robots.txt. It does not stop user fetchers, which may ignore it, or scrapers that hide their identity. robots.txt is a request, not a lock, so treat it as a signal.
The grey zone is real. On August 4, 2025, Cloudflare reported that Perplexity used undeclared crawlers to get around no-crawl rules and removed it from its verified bots list; the same day Perplexity disputed it, arguing that user-driven agents are not bots. Anthropic adds that blocking its IP addresses may not work reliably and that robots.txt is the way to opt out.
How can a CDN or firewall block AI crawlers by mistake?
Security tools block bots that look automated, and AI crawlers and even Googlebot look automated. A bot protection setting, a country block or an aggressive firewall rule can return errors or challenges to them while your robots.txt says allow. The page looks fine to you in a browser and is invisible to the crawler.
This happens more than people think. Cloudflare began blocking AI crawlers by default for new domains on July 1, 2025, asking owners at sign-up whether to allow them. Google's AI features page recommends making sure crawling is allowed both in robots.txt and by any CDN or hosting infrastructure. I saw the worst case on a medical clinic in Dallas–Fort Worth: its firewall was blocking Google itself. After the fix, its average position went from 13.5 to about 9; the details are in the clinic case study. Perplexity's crawler page gives a useful rule for firewalls: allow a bot only when both its user agent and its published IP ranges match, so fake bots stay out.
How do you check which AI crawlers can reach your site?
Open yourdomain.com/robots.txt and search for each bot name in the table. Then check your CDN or hosting bot settings, look at server logs for those user agents and the status codes they receive, and run a live URL test in Google Search Console. Errors such as 403 for real bots mean something blocks them.
This check is part of every technical SEO review I do, and the Perplexity SEO guide covers that crawler in more depth. For how crawl access fits the rest of AI visibility, see the GEO hub or the home page.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, which you control with its own robots.txt group. Check your robots.txt to confirm what you allow, and let OAI-SearchBot in if you want to appear in ChatGPT search answers and links.
Is Google-Extended a crawler?
No. Google says it is a control token without its own user agent; Google's normal crawlers do the fetching. You control it in robots.txt with its own group, and blocking it does not affect how you appear in Google Search.
Can I block AI Overviews with robots.txt?
Only by limiting Googlebot, which also affects normal Search. Google points to nosnippet and related controls to limit what is shown. Think it through before using them, because they also limit how your page appears in normal Google results. Most businesses gain more from being cited.
Do AI crawlers slow down my site?
They can add load. Anthropic supports Crawl-delay for ClaudeBot; for heavy traffic, rate limits at the CDN are safer than blocking search bots. Check bot traffic in your server logs before deciding what to limit and whom to let through.
Should I add every AI bot to robots.txt?
No. Name only the bots you want to treat differently. Unnamed bots follow your general rules. That keeps the file simple and easy to review, and you avoid blocking a search crawler you actually want to let in. Review the file once a year.
Want to see these guides first? Choose JJ SEO Colombia as a preferred source on Google, and my articles are more likely to show up for you in Top Stories.
Follow on GoogleAdd as a preferred sourceSources
- OpenAI, Overview of OpenAI crawlers
- Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity, Perplexity crawlers
- Google Crawling Infrastructure, Google's common crawlers: Google-Extended (updated October 30, 2026)
- Google Crawling Infrastructure, Google's user-triggered fetchers (Google-Agent, updated August 19, 2026)
- Google Search Central, AI features and your website
- Google Crawling Infrastructure, How Google interprets the robots.txt specification
- Cloudflare, Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large (press release, July 1, 2025)
- Cloudflare Blog, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (August 4, 2025)
- Perplexity, Agents or bots? Making sense of AI on the open web (August 4, 2025)