Which AI crawlers should I allow in robots.txt, and what does blocking cost?

Last updated October 14, 2026. Written by the Third Coast AI team.

A company that wants to be cited by AI engines should allow the search-facing crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot) and may separately decide whether to allow the training crawlers (GPTBot, ClaudeBot and the Google-Extended control token), because the engine vendors now split the two jobs across different user agents and blocking the wrong one removes you from answers without protecting anything, while the user-triggered fetchers (ChatGPT-User, Claude-User and Perplexity-User) fetch on a person's request and, by their vendors' own documentation, may not honor robots.txt at all. The costs are published: OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," and Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."

This page lists every agent by name with its vendor's own description, explains the three-way split between training, search and user fetches, puts a number on what blocking costs, and ends with a sample robots.txt you can adapt. Everything quoted comes from the vendors' published crawler documentation, fetched and read in October 2026.

Which AI crawlers exist, and what does each one do?

More than most robots.txt files account for, because each vendor now runs two or three agents with different jobs. The table below is drawn from OpenAI's crawler documentation, Anthropic's crawler documentation, Perplexity's crawler documentation and Google's common crawlers list, with the vendor's own wording where it matters.

Why do the vendors split training from search?

Because publishers asked for it, and because the two uses have different economics. OpenAI's crawler documentation is explicit that disallowing GPTBot "indicates a site's content should not be used in training" while OAI-SearchBot is the agent that decides whether you appear in ChatGPT search, and it publishes separate IP ranges for each (gptbot.json, searchbot.json and chatgpt-user.json). Anthropic's crawler documentation draws the same line: ClaudeBot is the training crawler, Claude-SearchBot "analyzes online content specifically to enhance the relevance and accuracy of search responses," and Claude-User fetches when a person asks Claude a question. Perplexity's crawler documentation says neither of its agents is used for training at all. Google's crawler overview describes three categories, common crawlers that "always respect robots.txt rules for automatic crawls," special-case crawlers, and user-triggered fetchers that act when "the end user triggers a fetch," and Google-Extended is a control token layered on the common crawl rather than an agent of its own.

The split means a company can hold two positions at once: allow the search crawlers so the engines can cite you, and disallow the training crawlers if you object to your content training models. We think that is a reasonable stance for a publisher and an unnecessary one for most commercial sites, since the training corpus is also where an engine's prior beliefs about your company come from when it answers without searching, and a brand absent from training data starts every un-grounded answer from nothing.

How much does blocking cost?

For search-facing agents, the vendors say the cost is total: no OAI-SearchBot, no ChatGPT search citations; no PerplexityBot, no Perplexity results; no Googlebot, nothing from Google at all. Cyrus Shepard's May 2026 synthesis of 54 studies scores URL accessibility at 9.5 out of 10, the highest of 23 citation factors, which is the measurement version of the same point. For training crawlers the picture is less clean, and BuzzStream's March 2026 study of news publishers that block AI bots found 82% to 88% of them still being cited, through cached training data and third-party mentions, so blocking GPTBot or ClaudeBot does not make a well-known brand disappear from answers; it stops future content from informing the model while leaving the engines free to describe you from what other sites say.

There is also a cost that no vendor documents, which is the CDN. Vercel's December 2024 crawler study measured GPTBot at 569 million fetches and ClaudeBot at 370 million in a single month, and bot-protection products that score traffic by volume and behavior routinely challenge or block those agents with a JavaScript check that, since the fetchers do not run JavaScript, they fail. We find this on roughly one audit in three: robots.txt allows everything and the firewall allows nothing, and the company has been invisible to ChatGPT for a year without a single line of robots.txt to show for it. OpenAI's crawler documentation says robots.txt changes take about 24 hours to be picked up, and a CDN rule change takes effect at once, so the fix is quick once found.

Should you block anything?

• Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot without exception if you want to be cited.
• Decide on GPTBot and ClaudeBot on principle, knowing it removes future content from training and does not remove you from answers.
• Leave Google-Extended alone unless you have a specific reason, since it governs Gemini training and grounding and nothing in Search.
• Do not spend time blocking ChatGPT-User, Claude-User or Perplexity-User; their vendors say they act on a person's request and may not honor the file.
• Check the CDN and firewall against each vendor's published IP ranges, and test with a raw HTTP request using the agent's user-agent string rather than trusting the robots.txt editor.

Can you show a sample robots.txt?

Yes. This is the shape we recommend for a commercial site that wants to be cited everywhere and has no objection to training, with the training agents listed so they can be switched to Disallow in one edit if the company's position changes. Adjust the Disallow lines for your own admin and search paths, and keep the sitemap line current.

User-agent: *
Disallow: /admin/
Disallow: /search

# Search-facing AI agents (allow to be cited)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Bingbot
Allow: /

# Training agents (change to Disallow: / to opt out of training)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Two cautions on using it. A robots.txt that names an agent and allows it does nothing if a wildcard rule higher up blocks a path the agent needs, since the most specific user-agent group wins and inherits nothing from the wildcard, so each named group needs its own Disallow lines for the private paths. And the file controls crawling, not indexing or snippets; for Google the AI Overviews page covers the nosnippet and max-snippet directives that decide what can be used in an answer once the page is crawled. Once the file is right, the next check is whether the text is in the HTML at all, which the JavaScript page covers, and the audit checklist puts both in order.

Frequently asked questions